Rants and Revelations

I’ve seen dozens if not hundreds of articles stating the completely obvious: If you make people change their passwords every 90 days, put in place complexity rules and checks to stop them reusing passwords, and make them change the password on 4 different systems, the end result will be that people will need to write down their passwords somewhere near their computer. So why hasn’t the company I work at gotten that message yet?

It’s bad enough that I have to use the password recovery feature on 2 of those systems because it’s evidently not the one I wrote down, but the wonderful little system I use for generating passwords I can remember doesn’t work if I have to keep changing it.

Your cow orkers are evidently neither gamers nor Penny Arcade readers, so when somebody sends an email to the group saying that it’s so-and-so’s birthday, and to come to the breakroom for cake, it’s not a good idea to do a group reply to say “THE CAKE IS A LIE”.

I just got a notice from Software Update that it needs to reboot my computer to complete the installation of an update of Safari. Since when is a web browser part of the operating system, Apple?

It’s bad enough you have to reboot when they update Quicktime, but this is just a little stupid.

Conversation at lunch today between R (been using emacs since 1981) and me (been using vi since 1987). It started off with him saying something about how he knew our latitude and longitude because he had to put it in his emacs config.

Me: When I’m looking for an editor, the ability to tell me sunrise and sunset times isn’t high on my list.
R: Emacs isn’t an editor, it’s an environment.
Me: Emacs isn’t an editor, it’s a lifestyle.
R: Yup. There’s straight, gay, and emacs.

At that point, I was literally laughing out loud (and I mean “literally” literally, not in the internet sense of “I smiled a bit”). I was lucky I didn’t have any food or drink in my mouth at the time.

The main loop query of my waypoint generator app is kind of hairy. And trying to do an “explain” on a typical query shows why it’s so slow.

explain SELECT a.id, c.pdb_id, internalid, a.type, name,
address, state, country, latitude, longitude,
declination, main_frequency, elevation,
b.category, chart_map, tpa, ispublic
FROM waypoint a, type_categories b,
id_mapping c
WHERE a.type = b.type AND
a.id = c.id AND
country in (’US’, ‘CA’) AND
(a.type in (’AIRPORT’, ‘VOR’, ‘NDB’) or
(category = 3 and (chart_map & 7) != 0)) AND
deletedon is null;
QUERY PLAN
——————————————————————————————————————————————————————–
Hash Join (cost=4442.82..19955.46 rows=34752 width=111)
Hash Cond: ((a.id)::text = (c.id)::text)
-> Hash Join (cost=4.83..12938.32 rows=34752 width=107)
Hash Cond: ((a.”type”)::text = (b.”type”)::text)
Join Filter: (((a.”type”)::text = ANY ((’{AIRPORT,VOR,NDB}’::character varying[])::text[])) OR ((b.category = 3) AND (((a.chart_map)::integer & 7) <> 0)))
-> Seq Scan on waypoint a (cost=0.00..10759.48 rows=72467 width=103)
Filter: ((country = ANY (’{US,CA}’::bpchar[])) AND (deletedon IS NULL))
-> Hash (cost=4.37..4.37 rows=37 width=15)
-> Seq Scan on type_categories b (cost=0.00..4.37 rows=37 width=15)
-> Hash (cost=2091.77..2091.77 rows=127777 width=12)
-> Seq Scan on id_mapping c (cost=0.00..2091.77 rows=127777 width=12)
(11 rows)

Adding indexes on country and type doesn’t help. There is still that nasty looking “Seq Scan on waypoint a” line. And also, another “Seq Scan on id_mapping c”, which I don’t understand at all because the joining column, c.id, is a primary key, so shouldn’t there be an index involved?

I’ve got a few ideas on how to use the spatial capability of PostGIS to improve that query, so I’m going to have to run a few tests. The first few ideas I’ve had aren’t showing major improvements in “explain”. It looks like the whole “type in … or ((chart_map & NN) != 0)” is going to force a sequential scan on waypoint no matter what I do. Hmmm.

Ok, I just added some indexes, and now the PostGIS version runs in 45 seconds. Phew! Man, I love “-d:DProf”.

In order to support a new product development I mentioned in an earlier blog post, I re-did my existing waypoint database as a PostGIS geographical database. I also added some foreign keys and some other cleanup that I’ve been meaning to do for a while. But obviously, I don’t want to support two databases, so today I’ve been converting one of my existing waypoint generator perl scripts to use the new PostGIS database instead of the MySQL database it was on before, but without any actual GIS functionality. And Houston? We have a problem. Doing a full US + Canada data load on the MySQL version takes about a minute and a half. Doing the same load on the PostGIS version takes twenty five minutes. Something tells me that I need to make some adjustments here.

The theme I have been using for a couple of years on this blog (Maple) didn’t work very well with Wordpress 2.5 and I can’t find an updated copy anywhere. Since I’ve hacked the shit out of it over the years, I decided it was time to make a fresh start with a theme that’s actually supported. So the first thing I did was copy the default theme, and try to make it look more like Maple. That wasn’t fun, and I’m not 100% satisfied with the results. So I decided to look at other themes.

I couldn’t find what I wanted at the official themes site, mostly because their search engine doesn’t categorize themes by category, and also because their “preview” function doesn’t work. But I found this other site, whose name I won’t mention but it had the word “free” in the URL. I found a bunch of nifty themes there, and downloaded them. I was just customizing one theme to add gravatar support and give the comments an alternating colour scheme, when I noticed something weird down at the bottom. It looked like spam. I grepped the theme code for the spammy urls, and couldn’t find them. But I figured the offending code must be in footer.php. Sure enough, all the themes I’d downloaded from this site had encrypted code in footer.php. I couldn’t read it or decrypt it, but it obviously was there to insert spam code in your blog. I tried replacing the footer.php with one from the default theme, and that broke other stuff. Crafty buggers.

Anyway, I’ve removed all traces of this crap, and I guess we’re all stuck with the psuedo-Maple theme until the official site starts working again.

I upgraded my blog to Wordpress 2.5 because the damn thing was nagging me all the time about being back at 2.3.3. But now I discover that my theme doesn’t work right with the new code, and one of my favourite plugins, the LiveJournal CrossPoster, doesn’t work. Now I’ve either got to find a less ugly theme, or fix the old Maple theme to support the new comment code with the built-in Gravatars.

Update: I might have found the fix for LJXP!
In lj_crosspost.php, change

if(version_compare($wp_version, “2.1″, “< ")) {
require_once(ABSPATH . '/wp-includes/template-functions-links.php');
}

to

if(version_compare($wp_version, “2.3″, “< ")) {
require_once(ABSPATH . '/wp-includes/link-template.php');
} else
if(version_compare($wp_version, "2.1", "<")) {
require_once(ABSPATH . '/wp-includes/template-functions-links.php');
}

Update #2:
I officially hate this update. It keeps adding bogus </code> tags even though my tags are perfectly well closed before I saved them. Let’s try with block quotes instead?

On March 25th, I ranted about a developer who had checked in a bunch of stuff that required a special “upgrade DVD” without telling the other developer that this would be required in Rants and Revelations » Developer dumbassedness. Well, it turns out it’s worse than that. Far worse. Not only didn’t he not tell us about the magic DVD, but he hadn’t even tested the damn DVD. It’s now 2 weeks later, and his DVD *still* doesn’t work.

So now I’m caught in a vice - I had to “rebase” my development environment to the new build in order to deliver some bug fixes, and now my development environment and my test system are at different builds, which makes it hard to test things, and especially hard to use the remote debugger in Eclipse. And I can’t get them back in sync until this damn DVD is ready.

I’d also like to mention that I did a much more ambitious upgrade DVD a few years ago, where his DVD is upgrading from CentOS 5.0 to CentOS 5.1 and reformatting the content partition from ext3 to xfs, mine was upgrading from RedHat 7.3 to CentOS 3.3. And I didn’t leave the rest of the developers out to dry because I tested the hell out of it on my test system, reformatting it back to RedHat 7.3 and running versions of my upgrade script over and over for weeks before I put it into the development stream.

I got an email from one of the sysadmins at NCF saying that the news directory has run out of space. After poking around a bit, I’ve discovered that:

• cron jobs, including the nightly expire job, haven’t run since March 18th
• I haven’t been receiving emails sent to the NCF news account, possibly for even longer than that, which is why I didn’t notice when the system throttled 3 days ago. Normally newswatcher sends these emails which I have forwarded to SMS so I don’t miss them.

The sysadmin wonders if the cron jobs not running has anything to do with the DST change. The machine is ancient, and running an ancient version of Solaris.

Of course, the fact that I didn’t notice the lack of the daily news admin email in my morning scan-and-delete folder isn’t good, either.

I probably shouldn’t give too many details, but I’ve been in talks with a certain freeware developer over developing a flight planning application for a web connected hand-held device. (Anybody who knows anything about me can probably guess the developer and the device.)

My part would be a server app that would respond to requests for data from the device and send new data or updates. Nothing too different than what I’ve been doing before, but one of the things we’ve been talking about is managing “areas”. His concept was that if the user entered an id that wasn’t on the device already, my server would send the device a whole “area”, and the device would keep track of what areas it had in memory already, when they were last updated, and would occasionally request updates of the areas it knew. He thought that each area could be a whole country. The first thing that struck me about that is that if the point you asked for was in the US, you could be asking for thousands of waypoints (70,584 in the current database). That could take a long, long time on an Edge network. Then we discussed maybe breaking it down by state or province in the US and Canada.

But the thing is, I used to be a GIS (Geographic Information Systems) programmer. I know there are better ways. At first I started looking around for the HHCode algorithm since I worked with Herman Varma and the Oracle guys implementing the original Oracle “Spatial Data Option”, until that scumbag Jim Rawlings screwed me out of three months pay. But I can’t find the source code anywhere.

So my next idea was a modified quad tree. Basically, when populating the database, I made a “rectangle” that incorporates the whole world and start adding points. When I hit a threshold, I subdivide that “rectangle” into 4 equal sub-rectangles, and move the points into whichever rectangle they belong to. This means that where points are sparse, the rectangles are large, and where they are dense, the rectangles are small. That way I’ve got some consistency in the size of the file to be sent to the device, and I’m not wasting people’s time sending the 19 waypoints in Wake Island, say, as an individual file.

I’ve been experimenting today with PostGIS, which is an extension to Postgresql which adds some very efficient geographic query tools. The program I wrote to take the data from my old MySQL database and put it into the PostGIS database while building these quad cells runs pretty fast. Surprisingly fast, even. PostGIS is pretty capable. Too bad the manual for it sucks rocks.

One thing that I keep forgetting is how much faster computers are now than when I was doing GIS for a living. I keep expecting things to take hours when they end up taking minutes, because the last time I did this sort of thing I was using a 40MHz SPARC and now I’m using a dual core 1.86GHz Intel Core2 Duo, and I’ve got more RAM at my disposal now than I had hard drive space back then.

Anyway, mostly I’m writing this because I’m really enjoying working with GIS-type stuff again. I wish I could do it full time again.

Today the developers were invited to see what our new usability expert has come up with. Evidently he hired some local company to do the graphics, and somebody else to whip it up into a fancy all-signing all-dancing Flash demo. It’s all eye candy and very little substance, and it looks childish to me. But evidently all the suits and managers love it, so it’s going to go ahead. I can’t tell you what it looks like, except the back drop looks like it was copied from the default background/splash screen/packaging of a certain fruit-based cat-themed operating system that was recently released.

The fact that the interface looks like it was designed more to impress suits than to help the people who are going to use it day to day isn’t the part that sucks. The fact that it’s all going to be written in Flash semi-sucks. The fact that it’s apparently going to be designed without talking to the people who’ve been working on the program for 6 years semi-sucks. What really sucks is that the project leader is talking about either outsourcing the entire Flash part of the user interface, or hiring their Flash programmer away from them. It was left to my cow orker Rohan to speak up and say “the reason you hired good people in the first place is that with a little training we can do anything, including Flash”.

Every morning, there is an ISO of the new build of our software in the drop box. If any of your code is new in this build, you’re expected to “integration test” your code to make sure it at least doesn’t make anything worse. Most people do it on the Integration Test Plex but some of us have our own mini-plexes cobbled together out of obsolete equipment.

This morning, I installed the ISO on my mini-plex as per usual. Only problem: my entire content directory was missing, and none of the software would run because it had evidently been compiled with Java 1.6 and the mini-plex has Java 1.5 installed on it. It turns out that one of the developers who doesn’t work with us very closely put together a new installation procedure that requires a special DVD instead of our normal installation procedure that is supposed to reformat our content directory as XFS, and upgrade it to CentOS 5.1. When I complained that nobody told us that we needed to follow a special upgrade procedure, he said “why didn’t you wait until it passed integration testing?”. Because I was trying to integration test it, dumbass!

After yesterday’s success upgrading Gallery, I decided it was time to upgrade the Browncroft Neighborhood Association CMS. It was running Drupal 5.1, and I see from the Drupal web site that the currently supported versions are 5.7 and 6.1. So first I tried upgrading to 5.7, and that worked fine. Then I tried upgrading to 6.1, and that failed horribly. Fortunately, both times I made a copy of the site on a test address in order to try the upgrade, which allowed me to try a few different approaches.

The upgrade to 5.7 didn’t require all the stuff they always tell you to do, like turning off all the optional modules and setting your theme to the default. The second upgrade failed silently a couple of times until I did it by the book and *did* turn off the optional modules. And that’s when I discovered that the upgrade script failed on some “ALTER TABLE” commands, which makes me think that it was expecting a newer version of PostgreSQL than the one I have installed.

I note that Debian Etch has both PostgreSQL 7.4 and PostgreSQL 8.1 packages available. I installed 7.4 because it was the default. But from work, I know 8.1 has more “ALTER TABLE” commands than 7.4. So now I’m trying to figure out just how I go about changing my 7.4 installation to 8.1 without breaking Drupal, Gallery, and all my custom configuration. So, lazyweb? Anybody out there do that upgrade and know?