Archive for January, 2006

A little while ago I set up a Wiki to discuss the imminent loss of the Digitial Aeronautical Flight Information File (DAFIF). Yesterday, the spammers found it. For two days now, some asshole has registered a new account and make his “user page” a spammer link farm. Now I could continue going in and ripping these pages out, but that could get really boring.

I think what I need is to replace the simple Wiki software I’m using (TWiki) with something with a bit more features. Specifically, I’d like to make it so that you can’t start editing until you give the software your real email address, like the way my mailman mailing lists require you to get and respond to a confirmation mail.

Any suggestions? (And yes, I know I asked for suggestions before and then ignored them all in favour of one that was easier to install. I’ll go back and read your suggestions from that time as well.)

I wrote a few days ago about a problem I was having with SQLite, or rather with DBD::SQLite. Turns out that one should never assume that the version you installed on one machine is the same as the version you installed on the other. After making sure the machine I was testing on was up to DBD::SQLite version 1.11, that part of it worked fine.

I’ve been doing some timing tests on the a generator task that generates 26915 waypoints, and doing one at a time it takes about 45-50 seconds and doing two at a time takes twice as long (about 1:30-1:40), as opposed to MySQL which takes 3:40 for one at a time, and 4:45 for two at a time. The fact that the SQLite one takes twice as long when there are two running makes me think it’s probably CPU-bound. The fact that it’s way, way faster than the MySQL alternative makes me think this is definitely worth pursuing.

But there’s a wrinkle. According to a post on the SQLite mailing list, one program can’t commit a write while another one is doing a query, even if the writes don’t involve the same tables. I guess SQLite’s database level locking is pretty stupid. But that’s the problem - there are three different types of things going on:

  • Database reloads - these only happen about once a month, only one at a time, and involve reloading FAA and DAFIF data into the waypoint, comm_freqs, runways and other similar tables. The reload scripts can take a hour or more to run.
  • Database generations - these run in the background, and just query those same tables that the reloads are loading. Lots of these run can run at once, and lots are run every day. As mentioned above, they only take a few minutes to run.
  • Choosing generating options in the web site. These tasks run after clicking one form page on the navaid.com generator and generating the response page. These mostly do some queries, but as well they track what options you’ve chosen so that they will be the defaults next time you come back. It does that by updating some tables which are not involved in the database generations.

Obviously a user doesn’t want to be sitting there waiting for their page to load for as long as it takes somebody else’s generation script to run. I’m going to have to try putting the tables that are used for storing these options in a different database to see if that will enable the pages to update in a reasonable time.

Ok, I found out why inn wouldn’t upgrade - I’ve been using “timecaf” for the news spool. This is a semi-binary format which is supposed to be faster and more efficient than “tradspool”, which is the old single file per article in a directory structure based on the newsgroup names that we all used to know and love. “timecaf” creates just a few files per day with multiple entries in each file. I forget why I stopped using “tradspool” because this machine is way overpowered, maybe it was to see if we could use it at NCF.

Timecaf has been working pretty well for me, but evidently it has binary file offsets embedded in the file. And RedHat (oh, sorry, the “Fedora Community”) arbitrarily decided to enable “Large File Support” in between Fedora Core 3 and Fedora Core 4. This means that each record in each “timecaf” file has a 32 bit file offset attached to it, but the program is expecting a 64 bit file offset. That makes it impossible for the program to find the records.

I tried a few things, including compiling the source without large file support, and I still couldn’t get it to work. So I threw in the towel and blew away the whole news spool. After all, this is usenet, every idea comes around again in a few weeks anyway.

The WordPress SpamKarma plugin was complaining about invalid SQL syntax. A little investigation showed that SpamKarma was expecting MySQL 4.0 or newer, but reluctantly allowed you to use MySQL 3.x as long as you accepted there might be some problems. I’d had other problems with the old versions of software that were part of Fedora Core 3.

I decided it was time to upgrade to Fedora Core 4. I don’t know for sure, but I think I’ve been on FC3 for a couple of years now. I would rather upgrade to Debian Sarge, since this machine is really just a server now, and I hardly ever use X Windows on it. But that would require too much work. It appeared I could upgrade from FC3 to FC4 using yum, with minimum downtime, so I set aside some time today to do it.

The upgrade is now finished. It was surprisingly smoothly, except for one thing. Inn won’t run. I can’t even rebuild the history files with makehistory - it dies with a SEGV. I’ve tried all sorts of things, and I’ve tried writing to Russ, the main developer. I have a bad feeling that I’m going to have to switch to installing from source, and I don’t want to lose the advantages that you get when you let somebody else (in this case, the Fedora team) manage the upgrades and dependencies.

It appears that the problem I’m having with DBD::SQLite (the SQLite Perl bindings) was reported as a bug to the maintainer 9 months and several versions ago and it still hasn’t been fixed.

#12674: DBD::SQLite ignores parameter bindings

Pity, since not being able to write the correct values is kind of a show stopper.

1. I’ve been playing around with SQLite on my linode. The database generation script runs twice as fast as when I use MySQL. But I’m having one major problem with the update script - it’s probably due to the Perl DBD::SQLite, rather than SQLite itself, but if I insert into a VARCHAR column using


my $wptInsertStmt = $conn->prepare(qq{
INSERT INTO wp_test
(id, datasource_key, tpa)
VALUES (?, ?, ?)});

$wptInsertStmt->execute('abc', '111', 555);
$wptInsertStmt->execute('1E2', '222', 123);

the ids that look like numbers get interpreted as numbers, so that ‘1E2′ gets inserted as ‘100′. Both MySQL and PostgreSQL versions of this code get it right. I suspect I’m going to have to convert the code to use bind_params, which will bloat the code and lead to problems if I get it wrong. It also appears I have to close all my prepared statements.

2. I hereby officially declare THE DEATH OF THE FLOPPY. I was using a floppy to transfer files between my development machine and my test machine. But the shutter on the floppy I was using suddenly stopped working. So I got another floppy, and after a transfer or two, I started getting I/O errors on the new floppy. So I grabbed another, and the same thing happened. That’s when I realized that when the spring sprung, it probably fell off inside one of my floppy drives and is still wreaking havoc on every floppy I used since. Unfortunately I couldn’t find my USB drive, so I had to beg one from a cow orker. Even if it is at home, I’m thinking of buying one or more new ones. They’re certainly cheap enough.

3. The LUGOR mailing list today is full of the plaintive crys for help from a guy who seems to be doing web design for a guy who doesn’t seem to have gotten the idea that a corporate web site should be about content, not craptactular toys. The first request was for information on how to put an animated train running around the borders of the web page. Evidently the boss is a dentist who has toy trains running around the office, and he thinks this would transfer well to his web site. The other, equally brain dead request was for putting automatically translated versions (ie Systrans or Babelfish, not human translated) of all of his web site up and automatically directing people from those countries to the “translated’ version. Can you imagine what a great image you’d give to people in non-English speaking countries subjecting them to Babelfish versions? Especially since you’d be automatically redirecting them there, even if they spoke English. I told him to look at Engrish.com and imagine people all over the world submitting his site to the local equivalent of that site.

The other day they “upgraded” our Lotus Notes to a “6.0 style”. Evidently I had been using a “4.x style” without upgrading it. Ok the new view looks more modern, but it’s slow. Even by Bloated Stoats standards it’s slow. About the only thing I like about it is that when you delete something, it actually goes away off the list instead of still being there but with an “X” beside it until you hit F9. The user interface still sucks like a really sucky thing.

One of the things I really hate about Lotus Notes is that it doesn’t just encourage “TOFU” (Top post Over, Full text Under), it makes it hard to NOT do it that way. I noticed an option on the “reply” button that says “Reply with Internet-Style History”. Dare I hope? Would this be a way to do trimmed quoted text with the reply text interspersed? Breathless with anticipation, I clicked it. And disappointedly, I exhaled. I ended up with the cursor at the top, followed by the reply to the reply to the reply, then the reply to the reply, then the reply, then the original text. And it was quoted with a single line of “>” quote marks, not multiple quote marks for the multiply replied to text.

Oh well. I still continue to expect miracles, I guess.

- Dental surgery sucks. I got a temporary crown this morning. Between the numb mouth, the screeching of the drill and the feel of things being ground away inside you, or the two people with their hands in your mouth, one of whom is constantly telling you to open wider as your fight with all your mental might to not bite them and tell them to get the hell away from your mouth, there isn’t much to like. It’s four hours now, and I’m getting the feeling back in my mouth, but this temporary crown feels like a foriegn object in my mouth. At least it’s not cutting into my tongue the way the fragments of tooth were doing all weekend, but I probably bit my tongue a few times while it was numb.

- Canadian voters suck. It’s almost as if they’re afraid of insulting politicians they used to like, so they keep voting for the PM who gets more and more unpopular, but wait for him to retire and then vote his replacement out of office out of anger at the one who just retired. They did it to Trudeau/Turner, Mulrooney/Campbell, and now it looks like they’re going to vent their anger at Chretien’s arrogance and corruption on Paul Martin, who seemed like a pretty decent PM for a change. I’m just hoping and praying that Harper doesn’t get a majority, because he would be a giant step backwards for social liberalism in Canada. Plus, Canada is the only member of the G7 with a balanced budget, so why vote in a guy whose going to fuck with that to give giant tax breaks to the rich?

It was a surprisingly nice day. Actually, I hadn’t noticed it was going to be a nice day until I was reading Information Echo, where he was opining how he was going to miss the flying day due to family commitments. I quickly hit F12 to bring up the Dashboard, where the Aviation Weather widget showed me that the only clouds forecast were high cirrus. Ok, not exactly sunny, but extremely good VFR throughout the region.
Continue reading ‘Nice flight’ »

I applied for long term disability insurance. Not because I anticipate problems, but because I don’t want Vicki and I to end up broke and homeless if I can’t work.

After review of your application, we regret to inform you that we are unable to issue a policy to you because of your longstanding history of multiple joint problems.

Because if there is one thing that would prevent me from typing all day, it’s chronic knee pain.

I’ve just discovered  lighttpd, a small memory footprint web server that supports all the stuff I currently use on my Linode - it’s got server side includes, fast cgi, simple virtual hosting, and mod_rewrite type redirection.  And Ruby on Rails supports it.  Ruby on Rails also supports SQLite, a small memory footprint SQL RDBMS.

Considering the small memory on my linode, I’m thinking that a transition from Apache and MySQL to lighttpd and SQLite might be just the ticket to improve the performance.  Which will be important if I start playing around with Ruby on Rails, because these “do everything including the kitchen sink” things tend to be memory and CPU hogs.

Now I’m trying to figure out how to test a transition.  I could install both of them on my home server and run it on another port.  I could install both of them on my linode and run it on another port.  Both of those options have the problem that it’s not a 100% accurate test, because I’d have to change the code to support a different port, and then change it back if/when I make it the true thing.  Another possibility is to buy another Linode for a few months and try it out there.

One of my teeth exploded this morning.  I went to the dentist and they poked around, took a ton of x-rays, and scheduled me for about a bazillion appointments.  They haven’t put anything on the hole, but it isn’t hurting much.

In other news, somebody from my pimp agency came by to see me, because the INS was asking for proof that I have a legal right to work in the US, so I had to show him my green card.  Not sure why it didn’t take the last time I showed it to them.  I was asking him about the current job market.  Not that I expect anything - my contract has been extended to the end of the year, but it never hurts to keep your eyes and ears open.  He said that right now, the hot technology is J2EE.  I’m not entirely sure how J2EE differs from all the Java Swing and RDBMS stuff I’ve been doing is, but I suppose I should do some research and become buzzword compliant.

Which is too bad, because in aid of my desire to get some more experience with AJAX I was planning to teach myself more Javascript and maybe Ruby on Rails.  There just isn’t enough time, nor enough room in my head.

I finally got up flying today.  It wasn’t perfect weather conditions - there were low scattered clouds around the airport, and a solid overcast up at 20,000 feet to deprive me of the sunny day I craved.  It was a bit hazy and there was a definite wind shear around 2,000 feet, with the consequent small amount of turbulence.  But it was warm enough that I didn’t have to pre-heat, and thanks to our getting cheap hangar space this winter, no messing around with wing and cabin covers.  And the weather is supposed to utterly suck this weekend, so it was now or “never”.
Continue reading ‘Friday the 13th: A good day to fly’ »

I’ve been waiting for months for good weather so I can go out and get re-acquainted with the sky.  It had gotten to the point where I was resolved to blow off work on the first sunny day.  But wouldn’t you know, today was a sunny day, but I had an important meeting at 10:00am, another one at 1:00pm, and a doctor’s appointment at 3:30pm.  Didn’t leave much of a gap for getting out to fly, especially considering the time it takes to get a plane ready before and put it away afterwards.  Even after the doctor’s appointment, there isn’t much time before the sun goes down, so I went to Best Buy to get my XM radio and iPod adaptor installed.

Tomorrow is supposed to be sunny as well, but there is supposed to be a strong windshear at 2,000 feet.  (I think that’s what Canadian weather briefers refer to that as a low level jet.)  That can mean difficult landings and lots of turbulence.  Not ideal conditions for a fun flight.  But we’ll see.

Last night I wanted to install Acrobat Reader 7 on my laptop.  Normally installing a program in Mac OS X is just a matter of downloading a .dmg (Disk Image) file, opening it, and dragging the application out of there to the Applications folder.  Sometimes the .dmg has a package installer in it.  So I wasn’t too surprised when I found that the .dmg file I’d downloaded from Adobe had a package installer in it.  But when I ran the package installer, it didn’t just copy some files from within itself to places on the destination drive.  Oh no, that would be too normal.  Instead, the package installer fired up a Adobe download tool that downloaded another freaking .dmg file!  And that .dmg file had a package installer that actually installed the Acrobat Reader.

Can somebody please explain why they wouldn’t just let you download the Acrobat Reader installer package, instead of making you download an installer installer?

Rone, you want to comment?