Category: Rant

At work I’m working on a way to upgrade our systems in the field from RedHat 7.3 and the 3.3 version of our software to CentOS 3.4 and the 3.4 version of our software, while preserving as much as possible of our current content and state.

I’m experimenting with custom install DVDs with my own kickstart file. It’s been a process of trial and error, mostly error, so I’m burning about 3 or 4 DVDs a day. Each time I have to go bother the guy with the DVD burner to make sure he’s not burning anything, then copy my ISO file over to his machine, eject his blank CD (he always keeps on in the drive in case he has to burn something when he’s not here), burn the DVD, and run over to put the blank CD back.

I put in a request for a DVD burner. They cost about $50 these days, or $60 if you need the dual layer, which I don’t. My boss approved it, and I suggested that I go out after work and buy one at Circuit City, but for reasons I don’t quite understand, we have to go through “IT Purchasing”, aka “Mordoc, Preventer of Information Technology Upgrades”. Mordoc is charging our department $129 for this burner, and I’m told I should have it in two to three weeks, by which time I should be finished this project and not need to burn DVDs any more.

If I had any need for an IDE DVD burner at home, I’d buy one, install it at work, and when I’m done with this project take it home.

I’m trying to count up how many different ways this is wasting the company’s money doing it this way, and I keep running out of fingers.

There are three components to my waypoint generator

• A set of scripts to load or reload some of the data when an update comes in.
• A set of scripts that actually generates the databases.
• A web interface.

All three components are written in Perl, and all access the same database. As mentioned previously, I’m using MySQL because PostgreSQL was too slow on the limited resources I have on my Linode.

Last night, I ran one of the load scripts, and while it was running I tried to access the web interface. The web interface start up accesses and updates a couple of “session information” tables, which the load scripts have no reason to access. So somebody tell me why the web interface startup timed out with the error:

[Wed Jun 08 22:08:37 2005] [error] [client 66.67.112.52] FastCGI: server "/config_backup/navaid.com/htdocs/CoPilot/index.fpl" stderr: DBD::mysql::st execute failed: Lock wait timeout exceeded; try restarting transaction at /config_backup/navaid.com/perl/WaypointDB.pm line 312.


Line 312 in WaypointDB.pm is a line that deletes from the table sess_main. And like I said, nobody else should be updating it. So why the hell should the fact that a load script is running cause a lock wait timeout on that table?

It turns out that that test I ran yesterday that showed that Linode was even slower in mysql than it was in Postgres? Well, it turns out that I’d left the “;host=mysqldb.gradwell.net” in the connect string, so instead of hitting my local mysql database, I was actually going across the Atlantic Ocean to hit a database at Gradwell. D’OH!

I switched to using the local database, and the time came down to a slightly more acceptable 7+ minutes, but I was still I/O rate limited much of the time. Then I switched to using another guy’s database on his Linode (much better provisioned than mine) and the time went down to about 3+ minutes, and I never hit my I/O limit even once. (Which makes me think that multiple generators running at the same time won’t slow to a crawl.)

I’m starting to think that I won’t be able to host my application on Linode at all. Here’s the results of my latest testing:

Database	Home	Gradwell	Linode
PostgreSQL	7:46		21:01
MySQL	0:32	1:01	42:40

The abysmal performance on the last run, MySQL on Linode, appears to be because I’ve hit some sort of I/O limiting that they do when people do too much disk I/O (i.e. swap).

I’m going to try the tests again on Linode but with the database hosted somewhere else – either at home or on my Gradwell server. Even if that works, I’m not sure what that will mean about my options.

Followup to Rants and Revelations » Bad news on the Linode front:

I ran the same generator task on Gradwell and the Linode. On Linode, it took 21m1.1s, on Gradwell, 1m0.5s. Kind of a huge difference, don’t you think? So I copied the database and code to my home machine, which has 1024Mb of RAM instead of 96Mb, and dual Athlon MP1800+ processors, and it still takes 7m46s.

So either Postgres is way slower than MySQL, or I’ve done something really wrong when I ported the code.

I guess my next move is to try the Gradwell MySQL code on my home server and see how long it takes.