Rants and Revelations

If you’re going to send a guy a survey to find out his impressions of your recruiting/interviewing methodology, you might not want to send it to somebody who has been waiting two months for you to pay his travel expense claim. Because he might just think that you’re a bunch of disorganized fuckwads. Just sayin’.

And a bit of further advice - you might want to make sure your expense spreadsheet actually prints out correctly using Google Docs and doesn’t require one to steal a copy of Microsoft Office in order to use it.

I took the Lance over to Batavia for service. It was only supposed to take a couple of hours, and I’d be back at work before 11:00. Of course, it didn’t work out that way.
Continue reading ‘A day well wasted’ »

Some years ago I got a set of Shape Files (much as I hated ESRI at the time, they did beat us (GeoVision) fair and square with an inferior product and superior marketing) for the provincial and territorial boundaries of Canada and wrote a short little C program to do a “point in polygon” to determine what province a particular point is in. It’s set up so that it parses the shapes, then sits there waiting for a lat/long pairs to come over a pipe and writes the province code back over the pipe. I write a special lat/long pair (-999/-999) when I want it to exit. I use it all the time when I’m loading waypoints. The program has continued to work while my waypoint generator moved from being hosted at home to a webhost (Gradwell.com) to a virtual private server (Linode) to a colocation box. Unfortunately, I just discovered that somewhere along the way I lost the source code. Right now it has a small bug in that when the program that opened the pipe to it dies, it starts consuming all the CPU instead of shutting down. I can live with that, but it’s annoying while I’m doing all this testing with buggy load scripts. That’s why I went looking for the source code.

Maybe I should write a new one just in case?

Yesterday I thought I’d try running my loading scripts on my home machine because it’s so much faster than my colo box. And it was running faster, and had gotten further than the still running script on my colo box when I got hoist by my own petard. I like to partition my systems up with different sized partitions for different major directories, and unfortunately, on this home machine, I only gave 100Mb for /tmp, not realizing that MySQL uses a temporary directory when you’re doing a long transaction. According to munin, the script filled up /tmp and died just a few minutes before I woke up this morning. So I’ve gone into /etc/mysql/my.cnf, changed the temporary directory to /var/tmp, and restarted it. /var/tmp has 7.6Gb free, so it should be safe.

I made some major changes to the way data gets loaded into my navaid.com waypoint generator database, mostly in the processing of the “combined user data”. Mostly, I wanted to make sure that if “Bob” provides me some data on Canadian airfields that includes communications frequencies but no runway data that it doesn’t wipe out the runway data from the dataset of Ontario airfields that “Alice” provided me last year, but only updates the data that has changed in the overlapping part of those two datasets. Add in the possibility that a waypoint might have changed identifier or been resurveyed so the location has changed a bit, and you can see that there are a lot of possibilities to consider.

Unfortunately, considering all these possibilities is time consuming. I’ve been testing these new scripts with a dataset from one person that covers the entire UK and some nearby locations in varying levels of detail, and another that covers Ireland in great detail, but which is unfortunately no longer being updated because the person who provided it moved. Running both datasets would be an overnight job. But now that I’m satisfied with the results of that, I decided it was time to reload the old DAFIF data though these scripts to get the combined user data exactly the way I want it. But this has caught a couple of bugs in the scripts, one of which only manifested itself after 36 hours or so of running. That one didn’t even give me enough information as to why it failed, so I had to add some “use Carp” and “use Data::Dumper” magic to my scripts and then I re-ran it and found the actual cause after another 36 hour run. I’ve been almost continually running load scripts all week. I’m hoping this run will be it, but I’m not sure.

Since my new home box is so fast, I’m thinking one possibility might be to do the load processing on it, and then just mysqldump it and bring the dump file up to the colo.

I belong to a YahooGroups mailing list for a club I joined. This morning, in quick succession, I got a message, then I got a message saying it was a correction to the previous message, and then I got 13 (so far) messages with the phrase “Message Recall Failure” followed by the subject of the first message, sometimes with some spam or anti-virus catcher stuff inserted.

A google suggests that the person who sent the first message decided to use an Exchange feature that allows you to “recall” or cancel a message you sent. Evidently the recipients who use Exchange servers got these “recall” requests, but didn’t like them for some reason, probably because the original had come through the yahoogroups server rather than the originator’s server. And Exchange had expressed its disapproval of the recall message by sending a message back to the yahoogroups list address, which went out to everybody on the list.

I’m just hoping those 13 messages are it, and we don’t get any more.

Thanks to Elizabeth for forwarding me this extremely interesting discussion of the Floyd Landis case: Editors’ Blog (EnvironmentalChemistry.com): Floyd Landis verdict update: still waiting and waiting and waiting…

There is some very interesting information in the links in the article, and in the discussion on the post itself.

This morning I was investigating a problem at a customer site where their Manager (a program I wrote most of) is hanging. Thanks to “kill -3 pid” I discovered it was building a tool tip. Thanks to a database query, I found that the tool tip it was building was trying to display 430,000+ identical date ranges. Hmmm, looks like we’re going to be a while, and probably a little short of screen real estate at the same time.

I did a few database commands to remove the 430,000+ records and put back one copy of it. (I’m sure somebody has a command that will do both in one step - I did a “delete” followed by an “insert” because I’m lame.)

The Field Engineer at the customer site wrote to ask if I could tell him the database commands in case this happens again. I politely declined. I told my boss that this would be somewhat like giving handguns and tequila to a bunch of 9 year old boys. Repeat after me:

I’m sorry, you appear to have totally hosed the database. You’re going to have to wipe the database and re-injest all your content. You’ll probably be up and running again in two days or so.

Thanks to a post by Skud, I’ve discovered Freebase, which is, as Skud described it, “crack for information nerds”. It appears to be exactly what I was hoping to develop or find for my aviation navigation data project - a flexible structured information agregator, almost like a wiki with user definable fields.

You probably can’t see much of it, because right now it’s in alpha and by invitation only, but so far they’ve imported all of Wikipedia and added some links between then, and people have written some demo applications.

Like many AJAX applications, it has a tendency to get the dreaded “a script on this page is running slowly”, but otherwise it’s pretty nifty.

A couple of months ago I was so happy when my friend Mike took me out flying in his float plane, as I wrote in Rants and Revelations » Splashy fun.

Today, Mike and his flying partner Dave crashed that plane and died.

This sucks.

Yesterday I wrote that I’d turned off the old machine for the last time. Well, it turns out I was wrong. I had put the mp3s in the wrong location, and had the permissions wrong, and so the backup script’s rsync deleted the mp3s from the wrong place and then couldn’t write them to the correct place. Which meant that this evening I had to turn the old machine back on, copy the mp3s back over (only having to reboot the old machine twice as it froze up under heavy load as it is prone to do) and then shut down the old box once again. Now that that’s done, I have to re-import my iTunes library, which will undoubtedly continue long after I’ve gone to bed tonight.

On the plus side, though, I managed to get the UPS software NUT installed and working. My new box doesn’t have a serial port, which means I had to use the USB driver for the first time. The kernel first identifies the UPS as a X Box controller (xpad), but eventually the newhidups driver figured it out.

The old box is shut down, hopefully never to be turned on again, and the new box is up and running with nearly all the old functionality. It’s very quiet in this office with that old beast off.

As well as the base install, and fixing all the Ubuntu stupidity (and a few bugs in the munin-node scripts), I had to install and configure all the following:

Apache2
Postfix
Postgresql
MovableType
Bind
user accounts
bogofilter
spamassassin
Root cronjobs
ssh tunnel
My cronjobs
trn
/etc/profile
Mutt configuration
munin
squirrelmail
iptables/ssh-black
frm/nfrm
ftp server
NFS export /mp3s

I still need to get nut set up to control the UPS, and hook up a printer and configure it. I’d also like to get Java and eclipse set up so I can do some work on it.

Right now iTunes is cranking away importing the “iTunes Music Library.xml” where I changed the old nfs mount point to the new one. It’s taking a while.

1. Ubuntu is set up to use anacron, but the /etc/crontab is set up to not run the /etc/cron.daily stuff if anacron is installed by surrounding the normal “run-parts /etc/cron.daily” with “test -x /usr/sbin/anacron || ()”. Which is unnecessary because there is already a “0anacron” file in /etc/cron.daily to update the anacron timestamp if it’s being run out of cron instead of anacron. This means that there is no way I can get the /etc/cron.daily stuff to run at night when the system is idle instead of at 7am when I’m reading my morning email. I had to change the runtime in /etc/crontab to 3am, and delete the anacron checking. We’ll see tomorrow if that helps.
2. Ubuntu installs findutils and slocate. findutils has an /etc/cron.daily job called “find” that runs updatedb. slocate installs its own updatedb and moves the findutils one to updatedb.notslocate. It has an /etc/cron.daily job called “slocate” that also runs updatedb. The slocate deb file moves /etc/cron.daily/find, but instead of doing something useful like moving it where it won’t get executed, or updating it so that it runs the findutils version of updatedb, it just renames it to /etc/cron.daily/find.notslocate. So slocate’s updatedb gets run twice every night. That’s pretty useless.

I’m getting the impression that Ubuntu, in spite of all the hype, is a pretty damn amateurish operation.

Remember Rants and Revelations » Ten Ways to get fired and/or arrested at work? Well, I got an email from the author of the WSJ article:

Hi Paul,
Thanks for the candid feedback. As it turns out, I think there might
be room for a follow-up story on things IT departments wish employees
knew. Want to take a stab at offering some thoughts on this? I’m looking
for specific tips - along the lines of the last story - highlighting
what people can do to keep themselves and their companies secure and
prevent legal and regulatory trouble. I would especially welcome any
horror stories you can offer that illustrate why these tips are
important.
Along with the note, can you confirm your full name, your location,
the full name of your firm and your title there?
Let me know what you think, and thanks, in advance,
Vauhini

Vauhini Vara
Reporter, The Wall Street Journal
vauhini.vara@wsj.com
415-765-8281 (desk)
206-423-3232 (cell)

Here is my response:

Things your IT Department wishes you knew:

1. IT policies are there for a reason, not just to make your life more difficult.
2. You may not remember it, but chances are your employee file contains a document you signed where you agreed to follow IT policies, and giving them the power to fire you, sue you and/or charge you criminally if you violate those policies.
3. If you bring in pirated commercial software from home, your company could be subject to fines equal to many hundreds of times the cost of the software, not to mention costly audits and lawsuits. Guess who is going to end up paying those expenses? Is it worth risking your house and your career in order to use your favorite word processor instead of the one the company provides?
4. Viruses and file sharing programs are often written by criminal gangs in order to turn your computer into a “zombie” that they can use to attack other computers, share illegal porn, send spam, or steal information from your computer without you even noticing. And pirated software you get from file sharing sites often has viruses built in. That’s one reason IT doesn’t want you running software they didn’t provide on your work computer.

Congress Approves Surveillance Measures | The Onion - America’s Finest News Source

I agree 100% with Ed Albaugh: “You won’t need to eavesdrop to hear this: I voted for you assholes because you said you were against shit like this.” (Well, in my case substitute “gave money to” for “voted for”.)