Rants and Revelations

Yesterday, I asked the musical question “Just how fucked am I?”. I woke up to the answer this morning - sometime around 3:30am my domU stopped working. Good bye web sites, good bye mailing lists, good bye picture gallery, good bye blog. Dammit.

I emailed Dave at Anexxa, and found out that when they’d moved to the new rack, they hadn’t labelled the ports on the managed power, so they couldn’t power cycle my machine. But they were going out to the facility at 1:30pm and could do it them. So I volunteered to come out with them. Probably just as well that I did.

The box power cycled fine, and I thought I’d tried the upgrade from there at the terminal, thinking maybe apt turned off the network connections just before it asked a question or something. They’ve done dumber things - remind me to tell you the story about trying to upgrade Morphix from an X console some day. No dice - it got to about the same place, but hung just as hard. I suppose if I really want to do the upgrade, I’ll have to try booting to the non-Xen kernel and do it from there, but for now I’ve marked the kernel packages as “don’t upgrade” and I’ll leave it alone until I have occassion to be back at the colo.

Unpacking linux-image-2.6.18-4-686 (from …/linux-image-2.6.18-4-686_2.6.18.dfsg.1-11_i386.deb) …
Done.

My colo box consists of xen.xcski.com, the dom0 which controls the others, and then xen1, xen2 and xen3 which are the domUs. Because it was way easier to do it this way, the dom0 is running Debian “etch” (aka “testing”), while the domUs are running Debian “sarge” (aka “stable”). The problem with using “testing” is that there are frequent updates, way more frequent than with “stable”). The problem with remote updates is if something fucks up, there isn’t any easy way to fix it. Usually that’s not a problem.

Today’s upgrades include a new xen kernel. But it says it’s installing a new kernel, leaving the existing one there. So it shouldn’t be a problem, right? Well, I was wrong. It downloaded the upgrades, then got to the “unpacking” stage and hung. I can’t ssh to the dom0. I can’t kill the upgrade. It’s not responding to the munin probes. The only thing I can think of is doing a power cycle and maybe scheduling a site visit. But the domUs are running fine. So why would I do anything drastic while the real meat of the colo box is still going fine?

I don’t know what to do. Wait and see, I guess.

I’ve heard commercial flying described as hours of boredom punctuated by minutes of sheer terror. Private flying, on the other hand, especially in winter, sometimes seems like hours and hours of work on the ground punctuated by a few blissful minutes in the air.

Yesterday, we had two missions in mind - we needed to get the Archer N9105X out to Batavia for its annual, and I wanted to investigate a month old report (but not a formal squawk) that the Lance was impossible to start. I’ve been meaning to look into that but the weather has either been low clouds and snow or high winds and bitter cold, so I haven’t been inclined to go to the airport. And neither has anybody else it seems - there has been almost no flying of club aircraft this winter. Not like last year when it seemed like every weekend was a good one for flying.

It’s a bit of a problem when you want to get two aircraft ready for flight but you’ve only got one pre-heater cart.

I should mention that we’ve got the coolest pre-heater cart in existance. It’s got the standard propane bottle, battery and Red Dragon heater, some fancy ductwork to duct the heat from the heater into the cowling of the aircraft you’re trying to heat. But it’s also got an electrical panel so you can plug it into the wall to keep the battery charged up, and it’s also got a connector for a Piper External Power plug so you can use the pre-heater cart to jump start airplanes.

Anyway, only having one of them is sub-optimal when you’re trying to get two planes ready at the same time. Especially when you’re dubious about starting both of them and jump starting an aircraft requires one person at the controls and another person to remove the external power plug and stow the pre-heater cart once it’s going. So this is what we did:

First I pre-heated the Lance. I dragged it out of the hangar (man, that plane is heavy compared to an Archer) and into the sunshine so that it would hopefully not get totally cold soaked while we pre-heated the Archer. Paul P is very new as the Maintenance Coordinator for the Archer and he had an email from the previous one saying that if you hooked up the external power with the battery master on, it would actually charge the internal battery, so we did that while pre-heating it.

Once it was pre-heated and had been on this “charge” for a while, we decided that we’d try to jump start the Lance, and then once it was running, Paul P would try to start the Archer and if that didn’t work I’d idle the Lance for a while to warm it up and recharge the battery, then shut down and jump start him. But it didn’t work like that - the Lance wouldn’t crank at all, even with the external power. The prop wouldn’t move far enough to kick through the compression of one cylinder. Ok, time for plan B. I’ll have to deal with the Lance later, but right now we’ve got to get that Archer moving.

I moved the pre-heater over to start pre-heating the Dakota while Paul was to get the Archer started and go ahead. We knew the Dakota wouldn’t be a problem starting because he flew it two days ago. But unfortunately, Paul didn’t know the first rule of winter starts, which is you start the damn thing as soon as you get the pre-heater off it, and then you do the cockpit preparation. Instead, he must have sat there for 10-15 minutes with the fin strobe going, which meant that marginal battery was using power to spin gyros and the engine was getting colder. No doubt he also had the radios going and was getting the ATIS and contacting clearance as well. So by the time he tried to start, he got one good spin, but it didn’t catch that time, and it didn’t have enough juice for a second spin. So once again, it was disconnect the pre-heater cart from one plane and drag it over to another. I jump started him, and he left almost immediately after, which confirms my suspicions that he’d used battery power to get ATIS and his departure clearance.

Anyway, the Dakota was warm enough, so I dragged it out of the hangar and started it. No problems starting, and I did my pre-flight cockpit preparation with the engine running and left. It’s kind of amazing that the Dakota has almost the same engine as the Lance, (it’s got an O-540 de-rated to 235 horsepower while the Lance has an IO-540 (the I stands for fuel injection) at the full 300 horsepower) but the Dakota turns over so easily while the Lance is a hard cranker even at the best of times.

I got to Batavia while Paul was just finishing up talking to Jeff Boshart about the squawks on the Archer, so I had almost no shut-down time there. We got up and going again, and I put on my foggles and flew an ILS for practice. It’s still nice how much more situational awareness you’ve got with the Garmin 530 there - Paul pointed out that we were heading to a cloud bank in a mile or two, but I pointed out on the 530’s screen that we were just opposite the FAF and we’d get turned 90 degrees very shortly. And sure enough, we got the turn almost as soon as I finished speaking.

When we got back, I discovered that although there hadn’t been any bookings for the Dakota early in the morning when I’d booked the Lance for the ferry flight, when we made the quick decision to take it, somebody had already booked it. D’oh! I guess I should have checked. He was waiting for us when we got there, and he was surprisingly good natured about it.

Anyway, after we got back, I decided that as Maintenance Coordinator I needed to do something about the Lance. I grabbed the battery tester and battery charger from the line shed and decided to try to charge it up. That’s when I discovered I had to remove 18 screws from an external access panel, and two screws from the forward baggage compartment floor, and 4 quarter turn fasteners on the battery cover just to get access to the battery. Before I started, the hygrometer was showing 0% charge - none of the balls were floating at all. After an hour or so, the hygrometer was showing 25% charge. Progress of a sort, anyway. I figured that the guy with the Dakota should probably be getting back soon, so I started pre-heating the Lance again as well as charging it.

The Maintenance Coordinator for the other Archer (39Z) showed up - he’d been planning to fly, but had gotten delayed so he wasn’t going to fly but wanted to check out the plane on the ground. I prevailed upon him to help me jump start it, and he agreed so I put the battery charger away, put back all those damn screws, and then disconnected the pre-heater heater pipes and hooked up the jump start cables. I jumped in the plane and tried, and dammit, it still wouldn’t crank with just the power from the pre-heater cart battery. However, the POH says that if you flip on the battery master, you get the power of the external power battery and the internal battery, so I tried that and it actually started. Woo hoo. Did my cockpit preparation with the engine running, and away I went.

I took it for an hour flight to charge up the battery. It was a great day for it, sunny and the air was still and smooth. I buzzed around the Finger Lakes, and practiced flying a DME arc using just the DME instead of that horrible “turn 10 twist 10″ method they teach when you’re an IFR student. Works pretty well, although you really need good situational awareness to make it work. At one point, just for the hell of it I tried a steep turn - of course that’s the only time in the whole flight that Rochester Approach felt the need to point out some traffic. And boy was he confused trying to give me a clock heading.

The Lance has a graphic engine monitor which also shows you some other facts, like the outside air temp and the oil temp. One of the things it shows you is the voltage level. Early in the flight I switched it to showing the voltage level and it was a nice 13.6 volts. Then I turned off the alternator master, and it quickly dropped to about 10.5 volts. Near the end of the flight I did the same experiment and this time the voltage only dropped to about 12.3 volts, so I think that proves that the battery was charging.

Afterwards, every muscle in my body was complaining about all the pushing and pulling of aircraft out of hangars, running back and forth to the line shed to get appropriate keys and tach books and battery chargers and the like, and pushing the start cart around, and just standing in the freezing cold waiting for batteries to charge and engines to warm up.

I was at the airport from about 10am to about 5:30pm. In that time, I flew for nearly two hours. The work to reward ratio isn’t what I would call optimal, but I’d do it again in a minute.

“Hello, I’m a Mac”

“And I’m a PC”

“Eh? What’s that? I’m a mainframe, you young whippersnapper.”

“And next time you try to make that joke, I’ll smack you across the mouth”

My colo facility contacted me on Wednesday to say that this weekend they’d be moving my machine to a new rack, and also that they’d gotten a new IP range and I had to switch over to the new range soon, but they’d let me have both IPs for the switch over.

So today my system suddenly went off the air. I was sort of expecting it, but I didn’t see any shutdown messages because they just three-finger-saluted it. After a couple of hours, I phoned for an update, and was told that they’d just powered it up. But it still wasn’t responding to pings, until I mentioned to them that eth0 and eth1 are in the opposite order than what you’d expect.

Once it came up, I tried to configure an eth0:1 using the new IP. That actually seemed to work on the dom0, so then I tried to do it on my domU. That seemed to work too. I was able to ssh into both ips on both the dom0 and the domU. So I thought I’d swap the domU ips around, so the new one was eth0, and the old one was eth0:1, which would make it easier to get rid of the old one when I don’t need it any more. So I changed it in /etc/network/interfaces and rebooted.

But then suddenly things started going pear shaped. The domU was refusing to boot with an error about being unable to find /dev/hda1. On the dom0, “ifconfig” would just hang. And then it stopped responding at all. Now I was in full panic mode. I called Annexa and Dave called the guys who were doing the rack move and convinced them to go back to the facility. I met them there, and found that my poor box wasn’t even responding on the KVM. We power cycled it, and found that it wasn’t starting the domUs, and also that while it started up eth0 and eth0:1, it didn’t start the virtual bridge interfaces (peth0, vif0.0, vif7.0, vif8.0, vif9.0, xenbr0). That’s not good. It appears that Xen doesn’t like the extra interface or something. So I got rid of eth0:1, changed eth0 to the new IP, and rebooted. This time, it started up and so did the domUs.

I was still having a bit of problem with my personal domU - it didn’t want to resolve. Evidently somewhere along the way I’d decided to remove this program “resolvconf” that is supposed to maintain your name resolution for you, and when I did it had replaced my resolv.conf with one that looks like it was copied from my home machine. So I fixed that and things sort of worked, but in spite of the fact that I had the old IP on eth0:1 it wasn’t answering on it.

So it looks like I’m up and running, but I can’t use the old IPs. So you’re not going to see this until your DNS cache updates and you see the updates I made over at zoneedit.com.

Every time my external USB disk disconnects itself (like it did after a short power glitch on Wednesday), I have to google the kernel message to see if it remounted with USB 2.0 speed or the slower USB 1.1 speed. I just can’t seem to keep straight in my head whether “USB full speed” or “USB high speed” is the good one.

Note to self: it’s “USB high speed”.

I think.

Restore-Habeas.org | Restoring the Constitution Act of 2007

A. Because he heard we’re getting a snow blower.

Today’s large snow fall was the last straw - we have in and bought a snow blower. Home Despot had them for 30% off, and an additional 10% off if you got their credit card. We got a 5.5hp 2 stage Ariens with electric start and multiple forward and reverse speeds, and with the discount it was less than the list price for the Yard Machines 5.5hp one I was originally looking at that didn’t have electric start and only one forward speed.

It’s still hard work to push through the snow, but it’s got to be better than shoveling another plow hump.

I set up “automatic domain renewal” so that I wouldn’t have to take any action, or indeed have to think about it, when one of my domains comes up for renewal. So why do you send me four identical emails within 10 minutes telling me that one of my domains is coming up for renewal? I don’t need to know, that’s why I told you to take care of it! Even one message would have been more than sufficient. But do you really need to blast one to the technical contact, one to the billing contact, one to the registrant and one to the email address on the account? What the fuck is the purpose of having a separate “billing contact” if you’re going to write every email possibly associated with the account about a billing issue. It’s called “billing contact” for a reason, fuckwads.

Update:Added the “Where is this body you want disposed of” picture.

We had a Energy Star energy audit of our house recently. One of the things the audit mentioned that there was some asbestos on one of the vent pipes in the basement, and that would have to be “abated” before they could work. Today I decided to take care of it. The pipe in question goes through “the scary room”, one of two dirt floored areas of the basement. Until today, I’ve never set foot in either one of them. There is no apparent reason why this particular pipe has been wrapped in asbestos - the portion of the pipe that is wrapped isn’t close to the furnace so it’s not very hot, nor is it in contact with insulation or something flammable. The register it serves is in the “breakfast nook”, which we suspect wasn’t part of the original build of the house.

I got advice from experts in the field who assured me that the sort of asbestos that forms into sheets isn’t the dangerous kind, but I should still take some precautions.

So armed with a tyvec painter’s coverall, dust mask, googles and rubber gloves, I entered the scary room to do battle with the evil asbestos. And that’s when I discovered that the asbestos wrap evidently held condensation or external water against the bottom of the vent, and now the vent pipe has rusted completely away on the bottom. But it’s fine at the top, which means there is a part between the bottom and the top where there are sharp edges and flakes of rust. So instead of just removing the asbestos wrapping, I ended up hacking out the whole rusty pipe, and stomping it flat to throw it away. And the sharp edges cut my gloves to ribbons, but at least I didn’t get any visible cuts in my skin. Unfortunately the unwrapped part of the pipe only had one hangar on the whole length, so it fell down without the added support on one end. And the “box” where it went into the register in the breakfast nook was also wrapped and rusty so I had to remove it as well.

It wasn’t until after I was done that I noticed the notice on the air mask that said it’s not for asbestos. Sigh.

Well, the whole thing is down now. Hopefully I didn’t do myself any damage doing it.

I wrote about some work I’ve been doing on the Waypoint Generator in Rants and Revelations » Getting there, still some collateral damage. In that, I said I wanted to do some more testing. Well, I did. I reloaded the entire DAFIF dataset. The test took 4 straight days to run, and that’s not including losing a day or so when my router lost its mind. And what this test told me is that the new algorithm for eliminating duplicate points is overzealous.
For instance, it classified two Canadian airports, CYEE Midland/Huronia and CNL8 Wyevale/Boker Field, as being the same. They’re actually nearly two nautical miles apart.

I was calling points the same if the types matched and they’re within 0.05 degrees latitude and 0.05 degrees longitude of each other. Unfortunately that is just about 3 nautical miles in the north/south direction, which this test has shown is too wide a net.

The problem is that I want to spot duplicates when a waypoint changes id, AND when they update the coordinates. I’ve seen places where they’ve updated the coordinates by half a degree, especially in the case of user-entered data.

I think what I’m going to have to do is trust that the coordinates aren’t going to change a whole bunch at the same time the id changes. So what I’ll do is call something a duplicate if it’s within 0.05 degrees if the ids match, but within 0.01 degrees if the ids don’t match. That’s less than a nautical mile, and it would be pretty odd to find two airports within a nautical mile of each other. (A lot less odd to find heliports or reporting points, unfortunately.)

Damn, this means another multi-day test run, unfortunately.

Today’s Dilbert cartoon feels oddly familiar.

I saw (and stole) a sig quote once that said “If your VCR is flashing 12:00, then maybe Linux isn’t the OS for you”. Today I had a similar experience.

I host a bunch of mailing lists on Mailman at my domain list.xcski.com. Some of them I run, and some, like the ones for the Linux Users Group of Rochester (LUGOR), I just host and let somebody else run. One of the things that Mailman does is automatically send out a monthly ‘password reminder’ - as well as reminding people that they belong to one or more mailing lists and giving them the password and url they need to know if they need to make changes to their subscription, it’s also a good way for Mailman to test if the address is still valid so it can unsubscribe invalid addresses automatically.

The monthly mailing list reminder looks like:


This is a reminder, sent out once a month, about your list.xcski.com
mailing list memberships. It includes your subscription info and how
to use it to change it or unsubscribe from a list.

You can visit the URLs to change your membership status or
configuration, including unsubscribing, setting digest-style delivery
or disabling delivery altogether (e.g., for a vacation), and so on.

In addition to the URL interfaces, you can also use email to make such
changes. For more info, send a message to the '-request' address of
the list (for example, mailman-request@DOMAINNAME) containing just
the word 'help' in the message body, and an email message will be sent
to you with instructions.

If you have questions, problems, comments, etc, send them to
mailman-owner@DOMAINNAME. Thanks!

Passwords for YOUR_EMAIL_ADDRESS:

List Password // URL
---- --------

followed by the name of each list you subscribe to, the password you use for that list, and the url to change your user options (or unsubscribe) for that list.

Frequently on the first of the month, I get an email or two from people who see these emails and rather than following the instructions and doing their own list maintenance, write to me and say “please unsubscribe me” or “please change my address to foo” or whatever, and I have a form letter that basically says “read the instructions and do it yourself”. But today I got a new variation:

It seems I have been added to this list by mistake. It looks as though this
address was found from the lugor mailing list. Please remove me from the
xcski mailing list.

My response was along the lines of “If you can’t tell the difference between a domain name and a list name, maybe Linux isn’t the OS for you.” Harsh, maybe, but hey, it amuses me and I get so few pleasures from mailing list hosting.