Month: March 2010

Oh, that isn’t good.

I was trying to tar a bunch of stuff off a USB backup disk onto the new machine, and it suddenly started throwing all sorts of errors and couldn’t read any drives, not even the root drive to find the shutdown command.

First thing I’m going to check is moving the drives around, because I accidentally put the two new drives in the third and forth slots instead of one and two, so I’m going to fix that. If that doesn’t help, then I’m going to just turn of the Adaptec RAID controller and try a software RAID. If that doesn’t work, I don’t know what I’m going to do. Probably return the hardware and start again.

The Frozen Few

After last week’s warm sunshine, March had to tap us on the shoulder and remind us all who was in charge. It was about 37 degrees, extremely strong winds (gusting 17-29mph), and off-and-on rain. But Stephen and Jim were going paddling, so I went as well. I was delayed by something at work, so I texted them to tell them that I’d be half an hour late, so they agreed to circle back and pick me up, for which I’m grateful.

It’s been raining for a day, and I guess that caused some snow melt, because the Genesee River is at least a foot and a half or maybe two feet higher than it was on Thursday. The flood conditions caused swirling water, tricky currents, and lots of floating debris. On top of that, the winds were kicking up high waves in the other direction. I was definitely out of my comfort zone in the Thunderbolt today, but that’s a good thing – I need to get better at boat handling with this boat before the ‘Round The Mountain race. I got lucky that Small Swells and Long Lake were pretty calm conditions last year, but I can’t count on that for every race.

We started off down river into the wind. The fast current was definitely overwhelming the effect of the wind, but some of the waves were coming up on to the deck of my boat. Mostly I didn’t have too many problems handling the swirls and waves. We were doing 7.5 to 9.5 mph on the way down stream. We paddled all the way from Black Creek down to the Genesee Waterway Center.

When we turned around, we suddenly had a pretty strong bunch of waves coming from behind. I was definitely getting a bit of a ride on the waves, but even so I was having trouble going as fast as 5 mph, and I was definitely feeling the challenge. We got out of the waves and into the slower water near the inside shore, and the speeds increased a bit. We did a lot of that sort of reading the river and picking our way up through eddies and the like, and then ferrying over to the other side when the river curved the other way.

On one of those ferries, there was a very strong wind from behind, and so I was surfing and dealing with the swirls and currents, and I had to stop paddling and brace at one point – and just as I braced, I got a terrible cramp in my left foot. I’d had foot cramping problems last year, most notably during the last 3 miles of the Armond Bassett race. The cramp at this time was really bad timing. That’s the one time today when I felt most like I was lucky not to dump. But I didn’t, and I carried on to the end in spite of the cramp.

It was only when I got out of the boat that the raw wind and cold really started to hit me. My hands felt like claws as I was trying to get my boat back in its protective bag and back on the roof rack. The fact that there was a snarl of fishing line trapped in the rudder didn’t help. But I got out of there relatively unscathed and my nice hot shower afterwards felt really good.

Total distance: 7.22 miles
Total time: 1:18.

Setting up new server checklist

Don’t mind me, I’m just recording what I’ve done so far in setting up my new box.

  • Ordered new server
  • Ordered new rails for server
  • Ordered two 1Tb drives for server
  • Installed drives in server
  • Discovered rails were the wrong kind for this server
  • Grovelled around the net and found the right type of rails, ordered them.
  • Installed Debian on the server.
  • Tried just blasting the entire backup of the old server onto the new one was a disaster, went to Plan B.
  • Discovered that i386 Debian works fine, except neither the Xen nor the Bigmem kernels boot.
  • Downloaded and tried to install ia64 Debian, only to discover that’s the one for Itanic.
  • Downloaded and installed amd64 Debian. Xen kernel working fine.
  • Installed and configured munin. Discovered smartd doesn’t work because I’m using an Adaptec RAID controller. Tried to install dpt-i20-raidutils, but they don’t seem to work either. Copied some third party munin Xen nodes from old box backup.
  • Installed sshd. Copied “authorized_hosts” from old backup, configured it to only allow public key authentication.
  • Configured the dom0 to take less memory. 96M was plenty on the old box, but this one didn’t boot until I increased it to 128M.
  • Make lvm disks for the domUs.
  • Copied one of the backups. Had to change the sxp file to specify the amd64 kernel, and copy the /lib/modules/*-xen-amd64 to the disk space. It boots, but for some reason it won’t start up the network.
  • Copied another backup. This time it booted the amd64 kernel just fine, but got a lot of errors on start up. But it did connect to the internet and stuff, so I’m not sure how critical the things that didn’t start up were. May have to try installing an i686 kernel and booting the xen instances with that.
  • The box rebooted spontaneously while trying to copy a lot of files over at once. Will have to try again without the memory restrictions (and maybe with the non-xen kernel). Will also have to make sure that it doesn’t do anything bad if one of the domUs is doing heavy i/o.
  • Tried again copying everything over with the non-xen kernel with 4Gb, and it still died.
  • Tried to disable RAID controller, didn’t work. So made 4 separate 1-disk “Volumes”, and go back to install Debian amd64 again.
  • Configured with /dev/sda with 2Gb /boot, 1Gb swap, rest available. /dev/sdb with 2Gb /, 1Gb swap, and rest available. Made “available” parts of two disks into an MD0 software RAID 1, then made that into a PV for LVM.
  • Overnight untarring of backups of mp3s and xen1 didn’t crash it. Woo hoo!
  • Installed sshd, copied config from old dom0, tested sshing in with a public key.
  • Installed xen stuff, and munin-node.
  • Untarred backups of xen2-3.

Next steps:

  • Copy the backups verbatim onto those disks, and hope like hell that Xen can boot them.

Long paddle

Today was brilliantly sunshiney and “warm” (41°F), so Doug and Mike and I went paddling. We went up the creek, battling the current and the “suck water” low water levels. Since I’m in my slow fat Looksha (the same boat that this time last year I thought was too tippy to venture out in the bay), I was kind of holding everybody up, but they waited for me. After Ellison Park dog park, Doug was ready to turn back, but in spite of the fact that I was the slow guy, I wanted to go on. We ended up going nearly as far as the rapids that I’ve never managed to get up stream from, not even in a plastic boat. We probably would have gone all the way to the rapid, but there was a guy fishing and we figured if we turned back two hundred yards early we could avoid two encounters with his line.

On the way back down, we encountered a couple of guys in plastic recreational kayaks coming upstream. You have to admire the determination, because they didn’t have races and training goals to meet, they were just out enjoying the day. Doug warned them about the new law about having to wear a PFD between November and May because one guy wasn’t wearing his, and I think they were a little cheesed off about it as they defensively told us that he only had it off because he was in the process of taking off his jacket. After we were out of earshot, Doug mentioned that in a very short time the trout season will start and there will be more game wardens and cops around.

A little while later, we passed a canoer heading downstream. Again, not a racer, just a guy enjoying the brilliant day.

After we got back to the dock, Mike had to get going, but Doug and I took a short paddle out into the bay to look around. There is a lot more open water than last time, but the water level is at least a foot down from mid summer levels, and there are a couple of dangers lurking just under the water. Doug suggested we go out in BayCreek’s war canoe and put a marker or extension on the ones we can’t move.

We ended up doing a total of 9.68 miles in 2:07:26. Not exactly speedy, but not exactly fast conditions either. I was reviewing my GPS data from last year, and while I didn’t have the GPS until April, it’s clear that I’m way ahead of last year on fitness. (Another thing that’s clear is that I wasn’t really good about stopping the GPS at the end of a work-out. Either that or I ended a lot of workouts by paddling at 40 mph along city streets.) Last year the first time I did a workout of longer than 9 miles was the 30th of May when I did 10 miles to prove to myself that I could finish the Tupper Lake 9 miler that was a week later. I didn’t go to the “Round the Mountain” race last year because I didn’t think I could manage 10.5 miles in the middle of May. This year, I’m pretty sure that won’t be a problem. Surviving the kind of waves they had last year, now that’s a different matter.

IT’S ALIVE!

The GoogleBox lives!


Yes, after 4 days of downtime, my illustrious yellow 1U server has been revived from the dead. After it died, I asked the colo provider to power cycle it, and they said it didn’t come back up. I asked them to yank the box, and I picked it up and brought it home. Expecting a power problem, I first tried yanking all the hard drives and memory, but even then didn’t get any beeps or other activity. So then, I tried yanking one of the CPUs. I must have gotten lucky, because removing the #2 CPU got me a couple of POST beeps, and when I put back the memory and the hard drives, it booted just fine.

I’ve had this box since January 2007, and the CPUs are dated from 2001, so I guess it’s time to replace it. I ordered a slightly newer box off eBay that has twice as much memory and 4 SATA drive bays. With two 1Tb drives, it will have much more disk space, but more importantly the empty drive bays mean that if I need to expand, I can add newer bigger drives when they become available. I’m considering using software RAID to mirror the two drives, because even 1Tb is bigger than what I have now and I’m not hurting for space. And with lvm, I can plonk in two new 2Tb drives when the time comes, migrate the volume groups to the new drives, and yank the old ones. All the remains now is to decide whether to build a new OS and get everything working on it, or just restore from a backup and continue the upgrade path.

While I had the box home for a few days, I took the opportunity to do a long-delayed upgrade from Debian “etch” to Debian “lenny”. I didn’t want to tackle that remotely because there was a significant chance (and it happened several times) that I was going to get it into the situation where I needed to intervene at the Grub stage, and I couldn’t do that remotely (because the cheap colo facilities don’t give you remote boot consoles like the expensive ones do.) The biggest hassle of this upgrade was that I had to do some messing around to get a console to appear, changing the boot options on the box itself, and also the getty lines in inittab of each of the guest “domU”s. The second biggest hassle was that I had to install “udev” on all the guests so that ssh could work. Also while they were home I took the opportunity to make a backup of the whole thing, including the guests that don’t belong to me. Normally I just back up my own. That should make setting up a new box a lot easier.

I got all the fixing and upgrading and backing up done early this morning. I brought my box over to the colo company office at 10:15. And waited. And waited. And waited. I had a “ping -a” running so I’d know as soon as it came back. And I waited some more. The business office is the other side of town from the colo facility, so I figured there would be some delay. The company that used to own those racks would let me go out to the facility with them, but these guys bought the business from that company and they’re anal about security and won’t let me go. Well, it turns out that their scheduled visit to the colo facility was at 10pm – nobody told me that, of course, until after I’d started panicing that they’d all gone home for the weekend without racking my box. But here it is – they racked the box, phoned me to say it was powering up, and now I’m connected again. Hallelujah!