Category: Rant

Darn it!

Some years ago I got a set of Shape Files (much as I hated ESRI at the time, they did beat us (GeoVision) fair and square with an inferior product and superior marketing) for the provincial and territorial boundaries of Canada and wrote a short little C program to do a “point in polygon” to determine what province a particular point is in. It’s set up so that it parses the shapes, then sits there waiting for a lat/long pairs to come over a pipe and writes the province code back over the pipe. I write a special lat/long pair (-999/-999) when I want it to exit. I use it all the time when I’m loading waypoints. The program has continued to work while my waypoint generator moved from being hosted at home to a webhost (Gradwell.com) to a virtual private server (Linode) to a colocation box. Unfortunately, I just discovered that somewhere along the way I lost the source code. Right now it has a small bug in that when the program that opened the pipe to it dies, it starts consuming all the CPU instead of shutting down. I can live with that, but it’s annoying while I’m doing all this testing with buggy load scripts. That’s why I went looking for the source code.

Maybe I should write a new one just in case?

Hoist by my own petard

Yesterday I thought I’d try running my loading scripts on my home machine because it’s so much faster than my colo box. And it was running faster, and had gotten further than the still running script on my colo box when I got hoist by my own petard. I like to partition my systems up with different sized partitions for different major directories, and unfortunately, on this home machine, I only gave 100Mb for /tmp, not realizing that MySQL uses a temporary directory when you’re doing a long transaction. According to munin, the script filled up /tmp and died just a few minutes before I woke up this morning. So I’ve gone into /etc/mysql/my.cnf, changed the temporary directory to /var/tmp, and restarted it. /var/tmp has 7.6Gb free, so it should be safe.

Sigh

I made some major changes to the way data gets loaded into my navaid.com waypoint generator database, mostly in the processing of the “combined user data”. Mostly, I wanted to make sure that if “Bob” provides me some data on Canadian airfields that includes communications frequencies but no runway data that it doesn’t wipe out the runway data from the dataset of Ontario airfields that “Alice” provided me last year, but only updates the data that has changed in the overlapping part of those two datasets. Add in the possibility that a waypoint might have changed identifier or been resurveyed so the location has changed a bit, and you can see that there are a lot of possibilities to consider.

Unfortunately, considering all these possibilities is time consuming. I’ve been testing these new scripts with a dataset from one person that covers the entire UK and some nearby locations in varying levels of detail, and another that covers Ireland in great detail, but which is unfortunately no longer being updated because the person who provided it moved. Running both datasets would be an overnight job. But now that I’m satisfied with the results of that, I decided it was time to reload the old DAFIF data though these scripts to get the combined user data exactly the way I want it. But this has caught a couple of bugs in the scripts, one of which only manifested itself after 36 hours or so of running. That one didn’t even give me enough information as to why it failed, so I had to add some “use Carp” and “use Data::Dumper” magic to my scripts and then I re-ran it and found the actual cause after another 36 hour run. I’ve been almost continually running load scripts all week. I’m hoping this run will be it, but I’m not sure.

Since my new home box is so fast, I’m thinking one possibility might be to do the load processing on it, and then just mysqldump it and bring the dump file up to the colo.

Another reason why Exchange is not my favourite mail server

I belong to a YahooGroups mailing list for a club I joined. This morning, in quick succession, I got a message, then I got a message saying it was a correction to the previous message, and then I got 13 (so far) messages with the phrase “Message Recall Failure” followed by the subject of the first message, sometimes with some spam or anti-virus catcher stuff inserted.

A google suggests that the person who sent the first message decided to use an Exchange feature that allows you to “recall” or cancel a message you sent. Evidently the recipients who use Exchange servers got these “recall” requests, but didn’t like them for some reason, probably because the original had come through the yahoogroups server rather than the originator’s server. And Exchange had expressed its disapproval of the recall message by sending a message back to the yahoogroups list address, which went out to everybody on the list.

I’m just hoping those 13 messages are it, and we don’t get any more.