Category: Revelation

Setting up new server checklist

Don’t mind me, I’m just recording what I’ve done so far in setting up my new box.

  • Ordered new server
  • Ordered new rails for server
  • Ordered two 1Tb drives for server
  • Installed drives in server
  • Discovered rails were the wrong kind for this server
  • Grovelled around the net and found the right type of rails, ordered them.
  • Installed Debian on the server.
  • Tried just blasting the entire backup of the old server onto the new one was a disaster, went to Plan B.
  • Discovered that i386 Debian works fine, except neither the Xen nor the Bigmem kernels boot.
  • Downloaded and tried to install ia64 Debian, only to discover that’s the one for Itanic.
  • Downloaded and installed amd64 Debian. Xen kernel working fine.
  • Installed and configured munin. Discovered smartd doesn’t work because I’m using an Adaptec RAID controller. Tried to install dpt-i20-raidutils, but they don’t seem to work either. Copied some third party munin Xen nodes from old box backup.
  • Installed sshd. Copied “authorized_hosts” from old backup, configured it to only allow public key authentication.
  • Configured the dom0 to take less memory. 96M was plenty on the old box, but this one didn’t boot until I increased it to 128M.
  • Make lvm disks for the domUs.
  • Copied one of the backups. Had to change the sxp file to specify the amd64 kernel, and copy the /lib/modules/*-xen-amd64 to the disk space. It boots, but for some reason it won’t start up the network.
  • Copied another backup. This time it booted the amd64 kernel just fine, but got a lot of errors on start up. But it did connect to the internet and stuff, so I’m not sure how critical the things that didn’t start up were. May have to try installing an i686 kernel and booting the xen instances with that.
  • The box rebooted spontaneously while trying to copy a lot of files over at once. Will have to try again without the memory restrictions (and maybe with the non-xen kernel). Will also have to make sure that it doesn’t do anything bad if one of the domUs is doing heavy i/o.
  • Tried again copying everything over with the non-xen kernel with 4Gb, and it still died.
  • Tried to disable RAID controller, didn’t work. So made 4 separate 1-disk “Volumes”, and go back to install Debian amd64 again.
  • Configured with /dev/sda with 2Gb /boot, 1Gb swap, rest available. /dev/sdb with 2Gb /, 1Gb swap, and rest available. Made “available” parts of two disks into an MD0 software RAID 1, then made that into a PV for LVM.
  • Overnight untarring of backups of mp3s and xen1 didn’t crash it. Woo hoo!
  • Installed sshd, copied config from old dom0, tested sshing in with a public key.
  • Installed xen stuff, and munin-node.
  • Untarred backups of xen2-3.

Next steps:

  • Copy the backups verbatim onto those disks, and hope like hell that Xen can boot them.

Long paddle

Today was brilliantly sunshiney and “warm” (41°F), so Doug and Mike and I went paddling. We went up the creek, battling the current and the “suck water” low water levels. Since I’m in my slow fat Looksha (the same boat that this time last year I thought was too tippy to venture out in the bay), I was kind of holding everybody up, but they waited for me. After Ellison Park dog park, Doug was ready to turn back, but in spite of the fact that I was the slow guy, I wanted to go on. We ended up going nearly as far as the rapids that I’ve never managed to get up stream from, not even in a plastic boat. We probably would have gone all the way to the rapid, but there was a guy fishing and we figured if we turned back two hundred yards early we could avoid two encounters with his line.

On the way back down, we encountered a couple of guys in plastic recreational kayaks coming upstream. You have to admire the determination, because they didn’t have races and training goals to meet, they were just out enjoying the day. Doug warned them about the new law about having to wear a PFD between November and May because one guy wasn’t wearing his, and I think they were a little cheesed off about it as they defensively told us that he only had it off because he was in the process of taking off his jacket. After we were out of earshot, Doug mentioned that in a very short time the trout season will start and there will be more game wardens and cops around.

A little while later, we passed a canoer heading downstream. Again, not a racer, just a guy enjoying the brilliant day.

After we got back to the dock, Mike had to get going, but Doug and I took a short paddle out into the bay to look around. There is a lot more open water than last time, but the water level is at least a foot down from mid summer levels, and there are a couple of dangers lurking just under the water. Doug suggested we go out in BayCreek’s war canoe and put a marker or extension on the ones we can’t move.

We ended up doing a total of 9.68 miles in 2:07:26. Not exactly speedy, but not exactly fast conditions either. I was reviewing my GPS data from last year, and while I didn’t have the GPS until April, it’s clear that I’m way ahead of last year on fitness. (Another thing that’s clear is that I wasn’t really good about stopping the GPS at the end of a work-out. Either that or I ended a lot of workouts by paddling at 40 mph along city streets.) Last year the first time I did a workout of longer than 9 miles was the 30th of May when I did 10 miles to prove to myself that I could finish the Tupper Lake 9 miler that was a week later. I didn’t go to the “Round the Mountain” race last year because I didn’t think I could manage 10.5 miles in the middle of May. This year, I’m pretty sure that won’t be a problem. Surviving the kind of waves they had last year, now that’s a different matter.

IT’S ALIVE!

The GoogleBox lives!


Yes, after 4 days of downtime, my illustrious yellow 1U server has been revived from the dead. After it died, I asked the colo provider to power cycle it, and they said it didn’t come back up. I asked them to yank the box, and I picked it up and brought it home. Expecting a power problem, I first tried yanking all the hard drives and memory, but even then didn’t get any beeps or other activity. So then, I tried yanking one of the CPUs. I must have gotten lucky, because removing the #2 CPU got me a couple of POST beeps, and when I put back the memory and the hard drives, it booted just fine.

I’ve had this box since January 2007, and the CPUs are dated from 2001, so I guess it’s time to replace it. I ordered a slightly newer box off eBay that has twice as much memory and 4 SATA drive bays. With two 1Tb drives, it will have much more disk space, but more importantly the empty drive bays mean that if I need to expand, I can add newer bigger drives when they become available. I’m considering using software RAID to mirror the two drives, because even 1Tb is bigger than what I have now and I’m not hurting for space. And with lvm, I can plonk in two new 2Tb drives when the time comes, migrate the volume groups to the new drives, and yank the old ones. All the remains now is to decide whether to build a new OS and get everything working on it, or just restore from a backup and continue the upgrade path.

While I had the box home for a few days, I took the opportunity to do a long-delayed upgrade from Debian “etch” to Debian “lenny”. I didn’t want to tackle that remotely because there was a significant chance (and it happened several times) that I was going to get it into the situation where I needed to intervene at the Grub stage, and I couldn’t do that remotely (because the cheap colo facilities don’t give you remote boot consoles like the expensive ones do.) The biggest hassle of this upgrade was that I had to do some messing around to get a console to appear, changing the boot options on the box itself, and also the getty lines in inittab of each of the guest “domU”s. The second biggest hassle was that I had to install “udev” on all the guests so that ssh could work. Also while they were home I took the opportunity to make a backup of the whole thing, including the guests that don’t belong to me. Normally I just back up my own. That should make setting up a new box a lot easier.

I got all the fixing and upgrading and backing up done early this morning. I brought my box over to the colo company office at 10:15. And waited. And waited. And waited. I had a “ping -a” running so I’d know as soon as it came back. And I waited some more. The business office is the other side of town from the colo facility, so I figured there would be some delay. The company that used to own those racks would let me go out to the facility with them, but these guys bought the business from that company and they’re anal about security and won’t let me go. Well, it turns out that their scheduled visit to the colo facility was at 10pm – nobody told me that, of course, until after I’d started panicing that they’d all gone home for the weekend without racking my box. But here it is – they racked the box, phoned me to say it was powering up, and now I’m connected again. Hallelujah!

A bad day paddling is better than a good day erging

Today was a gorgeous day, brilliant sunshine and warm, but a bit of wind. Doug, Bill and I met up at Bay Creek for a paddle on the creek. The water was a bit lower than last time, but it was more open. Doug’s Burn is in the shop getting fixed, so he paddled a much tippier Jet. Or is it the other way ’round? I can never keep the KayakPro boats separate – too many 17’3″ long boats, and their web site says one is 438mm wide and the other is 438cm wide, which probably is wrong. Bill showed up with a Romany sea kayak – his wife and daughter are in the Rough Riders club, so this is his secret other life kayak. Since he normally paddles a surf ski, I can’t blame him for paddling a more stable (and more importantly, one without a leaky drain in the cockpit) boat.

Doug was paddling a strange boat and was on a low intensity week, I’ve had sniffles and sneezes for a week, and Bill hasn’t been paddling for months, so we took it easy. On the way up, a Forge Racing canoer came paddling down, and barely managed to grunt out a response to our hellos. One of us made a remark about how unfriendly the Forge Racing guys, and wondered why Jason, a kayaker and a really friendly guy, hangs out with them. And a few minutes later, Jason came by in a canoe – and unlike the first guy, he had a friendly greeting for us and a smile.

We didn’t go all that far, and we didn’t paddle that hard. But it was a beautiful day, and we were in boats and with friends. And that’s what’s important.

More erging

I went erging again. I’m sure you’ll be relieved to hear I didn’t bring my video camera. The last few times I’ve done a long “distance” workout, I’ve done half an hour straight, then for the last half hour I’d spend 15-20 seconds having a sip of water, paddle normally for the remainder of 9 minutes, then paddle very hard for 1 minute, repeat, then sip, paddle for the remainder of 8 minutes, and go hard for 2 minutes.

This time I tried doing the “9 minutes normal, 1 minute hard” right from the beginning. Given that I’d done a fairly hard 25 minutes on a rowing erg yesterday, that was probably a mistake. The breaks were often a bit longer than normal, because the foot brace on Stephen’s erg kept slipping, as well as having a few iPod issues. Everything was ok, but at the end I only got one minute and twenty seconds into my last hard bit when I just had this weird “I don’t want to do this any more” feeling. I wasn’t especially exhausted, and I didn’t stop because I was tired. I just didn’t want to continue any more. It’s kind of odd for me to get 59:20 into a planned 60 minute workout and stop then. You’d think that close to the end you’d just keep going.

I’m getting some major calluses forming on my right hand (the “control” hand) at the base of my middle fingers, and a bit of an open sore on my left hand on the top of my thumb. I’m not sure where that sore is coming from.