ramblings

One of the features of the new platform that I am working on is the ability to automatically retrieve all new seismic activities before the end-user posts that there has been an activity.  This will make the data transfer appear instantaneous to our clients and make us look all-around awesome.

So I have a script designed to run whenever I need it to automatically get any new activities and save them to the local database waiting for the end-user.  I set it up to run every 15 minutes until I got the multitudes of old data into the new platform.  I’m talking ~170,000 activities, and each activity takes about 45-90 seconds to process.

So below is my simple cronjob command to have it run every 15 minutes 24/7/365.

*/15 * * * * php /var/lib/symfony/cron/runcron.php >> /var/lib/symfony/cron/test_cron_log.log

Till Next Time

I am making progress on my company’s new website.  I’m about 3 months in and will be launching the first version of the site in a few weeks.  So it was dismaying, to say the least, when I found that when I created the database schema for the symfony project I used the wrong data type for a few fields.

In the current platform these particular fields are stored in MySQL ‘text’ fields.  Simple, expands as needed, no problem.  I accidentally set these fields in the new schema as varchars.  My issue is that for most users, my varchar(32) field would be fine, but for that occasional user who uses lots of explosives & different delays, this would mean a loss of data.  Not something I am very keen on.

I googled a bit and could not find any information on doing a reconfiguration of the database fields in symfony and I figured it wouldn’t be too difficult to manually convert the field to a blob and modify all the accessing functions.  In the end, I was right!  Assuming you don’t make any mistakes it is remarkably simple to change a database field from a varchar to a blob.  You only need to modify two files and four functions.  So, without further adieu, I give to you the code to make a blob field accessible instead of a varchar field for symfony through propel.

(Original code is commented out for reference)

// lib/model/om/BaseActivityInfo.php
	public function getActivityExplosiveType()
	{
//		return $this->activity_explosive_type;
		$content = "";
		if(is_resource($this->activity_explosive_type))
		{
			while(!feof($this->activity_explosive_type))
			{
				$content .= fread($this->activity_explosive_type, 4096);
			}
			rewind($this->activity_explosive_type);
			return $content;
		}
		else
			return $this->activity_explosive_type;
	}
 
	public function setActivityExplosiveType($v)
	{
// this is the new code (copied & modified from setActivityExtra1) to set BLOB type		
 
		// Because BLOB columns are streams in PDO we have to assume that they are
		// always modified when a new value is passed in.  For example, the contents
		// of the stream itself may have changed externally.
		if (!is_resource($v)) {
			$this->activity_explosive_type = fopen('php://memory', 'r+');
			fwrite($this->activity_explosive_type, $v);
			rewind($this->activity_explosive_type);
		} else { // it's already a stream
			$this->activity_explosive_type = $v;
		}
		$this->modifiedColumns[] = ActivityInfoPeer::ACTIVITY_EXPLOSIVE_TYPE;
 
		return $this;
/*
//	this is the original code for when the field was a simple varchar
		if ($v !== null) {
			$v = (string) $v;
		}
*/
	}
 
	public function hydrate($row, $startcol = 0, $rehydrate = false)
	{
		try {
//   ... other field's code ...
 
//		this is the original hydration code for the varchar type activity_explosive_type
//			$this->activity_explosive_type = ($row[$startcol + 36] !== null) ? (string) $row[$startcol + 36] : null;
//		this is the new hydration code for the blob type activity_explosive_type
			if ($row[$startcol + 36] !== null) {
				$this->activity_explosive_type = fopen('php://memory', 'r+');
				fwrite($this->activity_explosive_type, $row[$startcol + 36]);
				rewind($this->activity_explosive_type);
			} else {
				$this->activity_explosive_type = null;
			}
//  ... other field's code ...
			$this->resetModified();
 
			$this->setNew(false);
 
			if ($rehydrate) {
				$this->ensureConsistency();
			}
 
			// FIXME - using NUM_COLUMNS may be clearer.
			return $startcol + 92; // 92 = ActivityInfoPeer::NUM_COLUMNS - ActivityInfoPeer::NUM_LAZY_LOAD_COLUMNS).
 
		} catch (Exception $e) {
			throw new PropelException("Error populating ActivityInfo object", $e);
		}
	}
 
		if ($this->activity_explosive_type !== $v) {
			$this->activity_explosive_type = $v;
			$this->modifiedColumns[] = ActivityInfoPeer::ACTIVITY_EXPLOSIVE_TYPE;
		}
 
		return $this;
 
	} // setActivityExplosiveType()
 
// lib/model/map/ActivityInfoMapBuilder.php
 
	public function doBuild()
	{
		$this->dbMap = Propel::getDatabaseMap(ActivityInfoPeer::DATABASE_NAME);
 
		$tMap = $this->dbMap->addTable(ActivityInfoPeer::TABLE_NAME);
		$tMap->setPhpName('ActivityInfo');
		$tMap->setClassname('ActivityInfo');
 
		$tMap->setUseIdGenerator(true);
 
		$tMap->addPrimaryKey('ID', 'Id', 'INTEGER', true, null);
// ... other field's code ...
//	this is the original code for the varchar type activity_explosive_type field
//		$tMap->addColumn('ACTIVITY_EXPLOSIVE_TYPE', 'ActivityExplosiveType', 'VARCHAR', false, 32);
// 	this is the new code for the blob type activity_explosive_type field
		$tMap->addColumn('ACTIVITY_EXPLOSIVE_TYPE', 'ActivityExplosiveType', 'BLOB', false, null);
// ... other field's code ...
	} // doBuild()
 
} // ActivityInfoMapBuilder

UPDATE:
I was doing some work with these form elements and realized that the Validators need to be modified as well. It’s rather simple, all you need to do is change the validator to remove any size limitations and convert the Validator to a sfValidatorString from whatever type of sfValidator it was. Code below:

// lib/form/base/BaseActivityInfoForm.class.php
...other code...
// original code
//      'activity_explosive_type'          => new sfValidatorString(array('max_length' => 32, 'required' => false)),
      'activity_explosive_type'          => new sfValidatorString(array('required' => false)),
// original code
//      'activity_explosive_make'          => new sfValidatorString(array('max_length' => 32, 'required' => false)),
      'activity_explosive_make'          => new sfValidatorString(array('required' => false)),
// original code
//      'activity_explosive_quantity'      => new sfValidatorNumber(array('required' => false)),
      'activity_explosive_quantity'      => new sfValidatorString(array('required' => false)),
...other code...

Till Next Time

The hard drive on my development server was failing last week so I had the fun job of getting the data off of it while I was still able to.  The process of getting all my data and installing Ubuntu 8.04.2 LTS on my new hard drive took the better part of two days.  After getting the server back to a solid state from which to build my LAMP stack on, I decided it would be a good idea to follow standard security procedures and install fail2ban to block any unintelligent hackers.  I think any hackers worth their respective salt can get around a basic implementation of fail2ban, but this is just my dev server and I don’t have any sensitive data on it, so I’m not too worried.

Today I was toiling away on my company’s website, updating, fixing, etc…  I was attempting to upload ~7 MB, 1312 files onto the server and my ftp client, filezilla, crashed.  I figured it was a one-time crash, so I reloaded filezilla and tried again, crash again.  I tried one more time just to make sure I was doing everything correctly.  Crash again.  I believe the issue revolved around the number of files I was transferring at once, so I tarballed the files and pulled filezilla back up and….couldn’t log into the server with filezilla.  After a moment of troubleshooting I realized I couldn’t log into the server via filezilla or putty (I sftp through port 22).  I was able to login, with my same username/password through putty AND filezilla through the public ip address though.

After digging through the log files and ifconfig to make sure my server wasn’t dying again I pulled up my iptables.  Yep, I was banned.  Apparently when filezilla was crashing there was some sort of login attempt made to the dev server and each login attempt failed.  So after my three attempts to upload my files, I was banned.  Lesson learned, always have two access methods to any server.  It makes troubleshooting issues much easier.

BTW, for reference the commands for iptables is:

list all banned ip’s

iptables -L -n -v
Chain INPUT (policy ACCEPT 78238 packets, 18M bytes)
 pkts bytes target     prot opt in     out     source               destination
 
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 
Chain OUTPUT (policy ACCEPT 83695 packets, 66M bytes)
 pkts bytes target     prot opt in     out     source               destination
 
Chain fail2ban-ssh (0 references)
 pkts bytes target     prot opt in     out     source               destination
   36  3456 DROP       all  --  *      *       10.0.55.4            0.0.0.0/0
   22  1732 DROP       all  --  *      *       81.208.51.90         0.0.0.0/0
41764 7846K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

list all banned ip’s for a specific program (fail2ban-ssh) with line numbers

iptables -L fail2ban-ssh -n -v --line-numbers
Chain fail2ban-ssh (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1       36  3456 DROP       all  --  *      *       10.0.55.4            0.0.0.0/0
2       22  1732 DROP       all  --  *      *       81.208.51.90         0.0.0.0/0
3    41764 7846K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

remove banned ip from iptables

iptables -D fail2ban-ssh 1[line-number]

Till Next Time

I don’t know what it is about me, but whenever I see an ad, I instinctively look for the small print to find the ‘gotchas’. It may have to do with my cynical nature when it comes to advertising, or it may have to do with my dislike of most advertising.  But it’s an almost obsessive drive I have to check out the fine print to see where the company is trying to pull the wool over my eyes.  

For the few times in my life that I’ve been in a ’sales pitch’ position I always felt a little dirty afterwards. I’ve never been really comfortable trying to persuade someone to buy/lease/use something without giving them the low-down. I’m much more comfortable discussing the positives AND negatives of an idea/product/item than actively avoiding the negatives aspect of said subject.   Now I don’t have a problem selling anything, just in the act of persuading the other person.  The only time I’ve felt differently is when I’m selling someone on myself, most usually in relationships.  

Most everybody’s been on both side of a relationship breakup.  The break-upper, trying to be gentle yet firm and not giving in to the sales pitch when the break-upee is fighting to stay together.  And I’ve been the one fighting to keep the relationship when the girl wants to end it.  I guess my normal objectivity goes out the window when relationship emotions get worked up.  But that’s can be a good thing.  When in a relationship, there should be no holding back.  

I’ve grown accustomed to the usual disclaimers on any advertising related to wireless cellphone companies.  But an ad on facebook caught my eye today for AT&T.  

AT&T Ad

I am amused by AT&T’s lawyers coming out of the woodwork to update their disclaimers to more accurately reflect the contracts that they require their customers to sign to get the best deals on hardware.  I’m used to the regular disclaimer “2 year contract required, early termination fees apply”.  Simple, sweet, to the point.  Then a couple years ago they added “3G not available in all areas”.   I understand, don’t want to misrepresent AT&T’s (notoriously bad) 3G coverage.  But now they’ve added “significant restrictions apply”.  Has hell frozen over?  Pigs flying? (Oh wait, their flu is flying around the world….bad example!).  I guess some of AT&T’s recent bad press has them getting more sensitive to the response that their data rates (correctly) bring out of people.  But I am overly glad that they are now at least hinting some of the (artificial) limits they are placing on their data plans. 

Maybe people will see enough these disclaimers to cause them to look more closely at their usage agreements with their wireless carrier and start to ask questions on why they can’t use their phone or netbook the way they were designed.  And start to demand that the large sums of money we are paying to the wireless carriers allow us to use their networks for our own purposes and not solely to increase their bottom lines. 

Till Next Time

A couple weeks ago I came across a query that I needed to write that would only retrieve the records that had the same date  as another table’s record.  Since I am using propel, I looked for a simple way to format the datetime field to just match the dates in the query so that only the appropriate records would be returned.  I googled for a long time, not reaching any solution.

I ended up with a stopgap measure to get me by until I could focus more time on the problem.  I iterated through the results and used php to compare the dates and drop the nonequal date records.  Today the problem came to a head where I could not move forward with my work until I solved the issue.  Since I’m using mysql and propel, I ended up creating a custom criteria using the mysql ‘DATE’ function.  I know this is suboptimal in that my query is now tied to mysql, but I really don’t expect to be changing databases soon, or ever.

Most of the solutions that I found online would not work with the DATE function for one reason or another (another = I couldn’t figure it out).  I finally found my solution in the symfony forums (can you believe it!?!?!) here.  For some reason google failed me on this one.  So out of the billions of pages online I had to find the one (1) that had my solution.  Luckily time_to_find_solution < losing_my_mind.  So the very simple solution is this:

$c->add(ActivityInfoPeer::ACTIVITY_EVENT_TIME, 'DATE('.ActivityInfoPeer::ACTIVITY_EVENT_TIME.')= DATE('.ActivitySeismoInfoPeer::ASI_EVENT_TIME.')', Criteria::CUSTOM);

Till Next Time

I had to do some looking to find how to do an ‘or’ statement in symfony with propel.  I found out at a google’s group here and the api documentation from propel is here.  Once I found the information it was surprisingly easy.  Below is the code I ended up using to select records either updated in the last two weeks OR activities that are delayed AND activities that the posting times are NULL (implying they are not posted yet).

$c = new Criteria();
$c->AddDescendingOrderByColumn(ActivityInfoPeer::ACTIVITY_EVENT_TIME);
 
$criterion = $c->getNewCriterion(
ActivityInfoPeer::UPDATED_AT, date("Y-m-d H:i:s", time() - 1209600), Criteria::GREATER_THAN)->addOr(
     $c->getNewCriterion(ActivityInfoPeer::ACTIVITY_DELAYED, 1, Criteria::EQUAL)
     )->addAnd($c->getNewCriterion(ActivityInfoPeer::ACTIVITY_POST_TIME, null, Criteria::ISNULL)
     );
$c->add($criterion);
 
$this->pending_activities = ActivityInfoPeer::doSelect($c);

Till Next Time

I am a current not-very-satisfied customer of comcast’s internet service.  I’m a staunch opponent to their ridiculously pathetic television service.  Months ago when I heard that clearwire was going to be launching and selling internet at a reasonable price I was intrigued.  When I heard that they were coming to Atlanta, not quite sooner, but definitely not later, I was excited.  

Last week I happened to see a clearwire truck driving down the road and started getting my hopes up.  Today I dropped by clearwire.com on a whim to check if it was available here yet and to my astonishment it was!  I immediately ordered my package and am eagerly awaiting it’s arrival so that I can cut the comcast cord.  

My current service with comcast is 768K/128K, the lowest package comcast has, only because I am currently living by myself and trying to save money wherever I can.  I used to have the standard 6M/2M plan with comcast and it was nice, I didn’t have any problems with it.  But when I reduced my internet speed, not just the speed decreased, the responsiveness died as well.  It seems like I’m on dialup sometimes.  Oh yeah, and it’s $40 a month, way, way, way too much for this crappy service.  I haven’t changed providers solely because I will be moving in a  month or two and will possibly have to cut my internet altogether in case I move in with someone who has internet already. 

So I am excited to be taking my money away from comcast and going with true 4G!

Till Next Time

Here’s another quickie post.  I have to create symbolic links occasionally, and enough time has passed in between the uses that I always forget the syntax to create the link.  I got this bit of code from here.  So, without further ado:

ln -s <Destination> <linkName>

So for instance, I need a link from /var/www/web to /var/lib/symfony/web, the following works:

ln -s /var/lib/symfony/web /var/www/web

And voila! Done

Till Next Time

In working on my new prod server I only have write access to the symfony folder and www folder.  This isn’t a big deal in actual development, just in backing up my files.  I have become a zealot when it comes to backing up.  Not just backing up files, but keeping a log of where the development stands for each backup.

For instance, before I launch the backend application in symfony, I do a backup and then I do another backup right after I launch it, before I make any modifications.  This seems to help me when I mess something up so bad that I cannot recover from my mistakes.

Since I only have access to the two folders that I am going to be backing up, I don’t want to backup the already backed up files, I would start to take up an exponential amount of space for each backup.  Did a quick google and found the solution here.

cp -r `ls --ignore=file1` destinationDir

Simple and it works.

Till Next Time

I’m an avid follower of many blogs, one of those is engadget.  I love hearing about the newest stuff that companies are coming out with and wish I had enough wit, wisdom and caffeine in me to work for them.

But anyway, this isn’t about engadget, well, only tangentially.  I got a Dell mini 9 delivered this week (which I ordered after hearing about Dell’s sale for $199 minis from engadget).  Today I read on engadget yet another photo frame that has wifi, a 7″ screen, 512 MB memory for $199.  Now, this is one of the much cheaper photo frames that are out there.  I don’t see why people will pay $200+ for an lcd that scrolls through pictures when you can buy a $200-300 netbook and just set up flickr, or picasa on it.

In addition to have a larger display, a netbook has more storage, a web cam (usually), a web browser and a battery!  If you had a couple netbooks lying around the house, you can just set them up as photo frames on shelves and you’re good to go!

I don’t see any major change in this scenario until e-ink makes an entry into the photo frame market.  Judging by the latest e-ink CE products lately the photo frame market is secure for awhile.  I believe it will at least be the end of 2010 before the e-ink color displays become attractive enough and within the realm of affordability to become mainstream.

Until then, I’ll be viewing my photos on my mini 9.

Till Next Time