Upgrades, Updates

I upgraded WordPress to 3.0, and to celebrate I switched to the new theme. I’m going to to want to replace that banner picture at some point, but otherwise I think it went ok.

I “racked” my new colo box at LogicalSolutions.net/RackWire.com. Of course the first thing I did when I got home was discover that I had 15 un-installed security upgrades, including the kernels. Oh oh. I went ahead and upgraded, in spite of the nagging voice telling me that this didn’t go well last time. And the first thing I discovered that when it upgraded the kernels, it didn’t keep all the tweaked settings I had in /boot/grub/menu.lst, not even in menu.lst~. I also discovered that when I backed up the system before I brought it over to the facility, I only backed up the root partition, not /boot. Dammit. I’m currently rsyncing some files from the old box to the new one, but when that’s done, I’m going to reboot the new one, and cross my fingers and hope it will work. If it doesn’t, I guess I’ll see how well LogicalSolutions takes to me visiting my box for a few hours tomorrow.

Update: I rebooted my box with the new kernel and it actually booted! Hurrah!

Lessons Learned, Mistakes Made, Where To Next?

After a day to think about what happened, and to talk to Mike F, I realize that didn’t have the full picture. That isn’t so surprising, because my ability to look around isn’t all that great at the best of times and when I’m scared shitless I can’t even turn my head to one side or the other for fear of it changing my balance. It turns out that Dan and the rest of the team wasn’t far behind me, and were looking out for me even if I couldn’t see them. Apparently Paul D had dumped in his ski and was having problems getting back in, and Frank was having some sort of difficulty as well. Even Dennis had dumped at some point. So I guess it wasn’t just me, the conditions really were semi hard.

As I mentioned in yesterday’s report, I think the root of my problems started with the dump right at the beginning. While I was near shore so I could get out of the water and dump the water out of the boat, and I dried off pretty quickly in the warm air, I also knew I wasn’t dressed for an extended swim. And dumping out in the distance we were away from shore during the paddle would have meant an extended swim. I also know I can’t get back in that boat in water. Without any bulkheads, the boat fills up with water and even if I could get back in the boat would be too unstable to stay upright.

So what should I have done differently? Obviously I knew I was nervous after the initial dump and felt even worse when Dan said to set course for the light house 5 miles away and almost directly across the swells. But should I have stopped then? I didn’t think so at the time, and I still don’t think it would have been the right decision. Dan has dragged me out of my comfort zone on several occasions, and I’ve learned a lot from it. I can attribute much of my improvement in paddling from some of those sessions. Yesterday, I persevered as best I could, but I just never relaxed. We passed a beach on the way up, and I was honestly thinking of going in there and hitch hiking back to my car. Maybe what I should have done was gone there and just practiced paddling up and down the swells for a while until I felt comfortable enough to try going across them again? Maybe in retrospect I should have stayed back at the original beach doing that. But I didn’t, and it’s too late to change it.

So what should I do differently in the future? First thing first, I need a new paddle shaft. I don’t know how long that’s going to take – that probably depends on whether Lars, the guy who used to be the Brasca rep in the US has any left over inventory. So before that, I need to find a paddle I can borrow without tearing my elbows apart. Secondly, I want to get a surf ski. I was the only guy out there yesterday who wasn’t in one, and Paul D’s troubles notwithstanding, it’s a lot easier to remount a ski than a Thunderbolt – and as Mike pointed out, sometimes just the confidence in your ability to remount it is enough to get you to relax and enjoy the conditions. Baycreek has Dennis’s old V10Sport for sale nice and cheap, but I really think I’m good enough to learn how to paddle a regular V10 (which is skinnier and faster), or at the very least the Ultra layup of the V10Sport (which is lighter than the Value). I actually went for a short paddle in a V10Sport today, and it didn’t feel any less stable than the Thunderbolt. Another thing I need to do is start carrying my paddle float and pump when I’m in the Thunderbolt on the lake. Maybe they won’t help, but they might make me feel more comfortable. I probably should dress warmer in cold water, although I hate being over warm when I’m paddling. I’m not sure what’s the correct clothing option for hot air and cold water, but I need to find something – possibly my Hydroskin shirt, with a pre-emptive dunk in the water before I start so I’m not overheating. Another thing I think I need to invest in is one of those rear-view mirrors you can attach to your hat brim that cyclists use. Being able to see what was going on behind me out of sight might have been a comfort to me when I was freaking out. Another thing is practice, practice, practice. I felt pretty uncomfortable out on the lake in the Looksha in the past, but with practice it got easier and easier. I know that I’m going to get there with the Thunderbolt, and the sooner I can get back out there practicing, the sooner it will come. And the most important thing, though, is to have more confidence in the people around me. I know Dan can get distracted when he’s busy with the other guys, but I should have known that Dan and Mike and the other guys weren’t too far away and would have come quickly if they’d seen me dump. I was pretty hard on Dan yesterday, and now that I’ve had some time to calm down, reflect, and talk to others, I feel bad about that. So Dan, I know you read my blog, and I just want to say I’m sorry for saying you abandoned me.

Not my best day paddling

Today the team met for our first paddle on the lake. Originally Stephen had said I might be able to paddle his v10 sport surf ski, while he used his new v12. But it was too rough, so he took the sport, leaving me in the Thunderbolt. This is my first time on the lake in it, and probably only my second time in waves – the race being the first.

I was moderately nervous as I went out a few hundred metres into the swell. I waited for a small lull and turned down, and actually got a few nice runs. But as I attempted to turn up swell again, I dumped leaving me cold, wet, and even more nervous about the swell.

Dan immediately set course to a lighthouse which is about 5 miles away, but of course it was exactly 90 degrees to the swell, which is the worst. I was determined to give it a chance, hoping my nervousness would abate. I told several people how nervous I was and Mike and Paul kept fairly close to reassure me.

Dennis, a paddler I know through Facebook whom I’ve never met in person before, showed up. He was test paddling a new v10 sport to see if the cockpit was easier on his back than his old v10 sport. He, like everybody else except me, was showing every sign of enjoying the swells.

After half an hour of slow paddling, I realized that I wasn’t getting any less nervous, and the fear was making me more tired than hard paddling would. I was thirsty as hell, but couldnt grab a drink because i didnt dare take a hand off my paddle I knew I was never going to survive another hour of this. So I told Dan I needed to turn back and I needed somebody to go with me. He turned the whole group around. He paddled with me and said he’d stay with me. I was grateful because I’d already dumped once, in shore, and now we were hundreds of metres off shore and a long way from where we’d started. I knew that a dump out here would require a long, cold swim to shore, abandoning my boat and paddle since I am unable to remount this boat, and then a several mile walk back to the parking lot.

So it was with more than a little consternation that not more than five minutes later I realized that Dan wasn’t anywhere around. Mike was still with me, so that wasn’t too bad. He stayed with me until we were about a mile and a half from the start, but then he disappeared as well. Not sure where he went, but he’s not the one who promised that he’d never abandon me. Dennis came up beside me and talked to me. I was getting more and more tired, and the fear and the tiredness was making me less and less in control of the boat.

Dennis disappeared for some reason, and I just about freaked out. I was shaking, and on the verge of tears. I was sure my only hope was to swim for it. I screamed for help, and Dennis showed up – I think he’d just dropped behind. I was even less able to think straight or paddle straight. I was barely moving at all, and doing more bracing than anything else.

As I got parallel to the beach where I’d launched, I was able to turn down swell and pick up the pace. I didn’t quite get a ride from the waves, but at least I wasn’t fighting them. But as I was getting out of the boat, a wave hit me and I fell on my paddle shaft, snapping it in two. What a perfectly horrible way to end a perfectly horrible day.

At this point, I’m not even sure I want to replace the paddle. This was supposed to be fun. Maybe I’ll feel better tomorrow, but right now I don’t want to paddle and I definitely don’t want to see or talk to Dan.

Can’t win, shouldn’t even try.

Early on in my attempts to become a kayak racer, I encountered some elbow pain and got very discouraged. But after a while it settled down to a level that I could manage by taking way more Alieve (Naproxen Sodium) than any doctor would recommend. (Yeah, my kidneys will probably explode some year, but unlike chronic joint pain, doctors know how to treat that.)

I’ve been paddling a hell of a lot, and so far I’ve had no increase in elbow pain over last year – if anything I’m taking less Alieve than before.

But I’ve been paddling the same paddle, a Basca VIII-max, since I first learned how to paddle a wing paddle. It’s a good paddle, but very large. I’ve lately been wondering if there might be some advantages to a smaller blade. One thing about the VIII-max is that it’s so long it doesn’t get “buried” entirely in the water at the catch, but gets progressively more blade in the water as the stroke progresses. I consider that’s probably a good thing for my joints, but experienced paddlers like Dan and Jim keep nagging me about it. So I borrowed a Basca IV clone (I think it’s a Danny Broadhurst, but it doesn’t have any identifying marks on it) from Dan, and I paddled it yesterday for about 15 km. Jim kept telling me my catch looked very strong.

Today my elbows are sorer than they’ve ever been, and my shoulders are in agony. I’ve never had shoulder problems before. This is not a good thing – so far in my life, every time I’ve developed a new joint pain it has not gone away or even abated, ever. Starting with knee pain 35 fucking years ago. Every pain only gets worse. So I view a new joint pain with some alarm.

Needless to say, I won’t be paddling the IV again. I just hope that my shoulders forgive me when I go back to my nice gentle catch with the VIII-max. Mean time, I’m taking an unplanned day off of paddling, and gobbling the Alieve and Tylenol.

Another try at setting up the new server

  • Discovered that one of my hard disks was flakey and returned it. That’s probably why all my previous attempts to set this up failed.
  • Removed the daughter card RAID controller. The built-in RAID controller still sees the disks, but reports them at a JBOD (Just a Box Of Disks).
  • Started a new Debian installation.
  • Set up the both whole disks as the software RAID1 (instead of just a partition on each disk like I did last time).
  • Make the whole RAID (md0) into a physical volume (xen-space) for the LVM.
  • Created a 4Gb root partition and a 1Gb swap partition as logical volumes on the physical volume.
  • Did a base install. Noted that because I used software RAID on the whole thing, it uses LILO instead of Grub. Oh well, you can’t have everything.
  • Rebooted and the BIOS only saw one of the two disks.
  • Fiddled with the disk sled, rebooted, and this time it saw both.
  • Evidently the first boot without the second disk caused the raid to degrade, so re-added the disk mdadm /dev/md0 --add /dev/sdb1 and now it appears to be rebuilding.

Day 2:

  • Installed smartmontools, and enabled in /etc/defaults/smartmontools. Express slight concern that /dev/sda has an exit status of 64 because of some error in the log, probably due to the late unpleasantness. Will have to figure out how to clear that.
  • Installed munin-node and munin-plugin-extras, and copied the configuration from my backup from the last time
  • Installed openssh-server (unselect xauth which gets added automatically because it drags in a ton of X11 libraries). Copied /etc/ssh/sshd_config and /root/.ssh directories from backup.

Day 3:

  • Installed xen-utils. Holy shit that dragged in a lot of dependencies, and it said it had to “reinstall” 200+ packages for some damn reason. But then it gave an error, and when it came back it didn’t have to reinstall them after all. Very odd.
  • Didn’t see any xen in /etc/lilo.conf, so installed linux-image-2.6-xen-amd64. (Had originally thought that installing xen-utils would do that, I thought it did last time.)
  • Lilo complains that /vmlinuz is too big. According to the docs, lilo and xen don’t play together well, and grub has trouble with /dev/md0 software raid. I think I may have to go back to the drawing board, either re-installing the raid card, or going back to the primary boot partition and putting the software raid on the rest of the disk. Or maybe I can figure out how to get grub working. Once again I’m reminded of “Three Dead Trolls In a Baggie” singing “yeah, but I’ve got a girl friend and things to get done”.

Day 4:

  • Reinstalled the Adaptec RAID card, and set up a hardware RAID-1
  • Partitioned the “drive” with three partitions, one 4G ext3 for /, one 1G swap, and the rest as a physical volume for a lvm.
  • Installed on /, and when it went to reboot it got to “shutting down md0” and then hung. Will have to check that again. But at least it installed Grub instead of LILO.
  • After it booted, tried the “reboot” command and it worked! Yay!
  • Installed smartmontools, but discovered (once again) that it doesn’t work with the raid controller, so uninstalled it. I need to find if there is some other way to monitor the raid controller. I think I tried the dpt_i20 thing before and it didn’t work.

Day 5:

  • Installed sshd, copied the configuration from the backup to only allow public key logins. (Bite it, password guessers)
  • Installed munin-node
  • Installed linux-image-2.6-xen-amd64 and xen-hypervisor-3.2-1-amd64
  • Rebooted and the damn thing spewed tons of errors and hung. Tried to reboot with the old kernel (that worked before) and I got the same errors. I guess it’s time to give up on that hardware RAID again.

Day 6

  • Ran the disk “verify” tool in the raid card, and it didn’t find any errors.
  • Anything I tried to boot the system (the original kernel that worked before, single user mode) still failed in aacraid.
  • Ripped out the raid card again, and installed with /, /boot, /var and swap as primary partitions, and the rest of the space on both drives as a software RAID-1 used as a physical volume for LVM.
  • Install openssh-server (and unselect xauth). Copy /etc/ssh/sshd_config and /root/.ssh from backup.
  • Install smartmontools and enable it in /etc/default/smartmontoolsctl.
  • Install munin-node.
  • Rebooted to make sure everything starts correctly.
  • Installed linux-image-2.6-xen-amd64 and xen-hypervisor-3.2-1-amd64
  • Reboot again.
  • Ok, it booted, but “xm list” isn’t up.
  • Manually start xend and “xm list” is working.
  • Rebooted, and this time “xm list” is working.
  • Started to create the lvm logical volumes for the domUs

Day 7:

  • Discovered that when I backed up the last nearly successful domU, I forgot to back up the boot partition, so I’m on my own for the grub configuration.
  • Untarred my backups of the “xen2” and “xen3” domUs. Got a bunch of kernel messages about kjournald being blocked for more than X number of seconds while that was going on – I assume that’s because I was running up load averages in 7 and 8 range in the dom0, which is probably not a normal thing. I hope that just because things weren’t written to the journal immediately that doesn’t mean they were written wrong, only that I might have been in danger if things had died in the middle.
  • Installed rsync so I can restore my backup of the “xen1” domU.
  • Installed vim and removed vim-tiny
  • Restored backup with rsync --delete -aSurvx --numeric-ids /mnt/usb0/xen1/Sun/ /mnt/xen1/
  • Copy the amd64 kernel modules to the domU’s /lib/modules. cp -rp /lib/modules/2.6.26-2-xen-amd64 /mnt/xen1/lib/modules Must remember to exclude /lib/modules when I do any final rsyncing from the live domUs.
  • DAMMIT! It appears that I made /var too small again. Once it saves /var/lib/xen/save in it, the file system is full. Need to move things around again.
  • Booted into rescue mode, and moved things around. Everything seems to work now.
  • Try to rsync some newer backups.

Further updates as things progress.