Month: April 2010

Another try at setting up the new server

  • Discovered that one of my hard disks was flakey and returned it. That’s probably why all my previous attempts to set this up failed.
  • Removed the daughter card RAID controller. The built-in RAID controller still sees the disks, but reports them at a JBOD (Just a Box Of Disks).
  • Started a new Debian installation.
  • Set up the both whole disks as the software RAID1 (instead of just a partition on each disk like I did last time).
  • Make the whole RAID (md0) into a physical volume (xen-space) for the LVM.
  • Created a 4Gb root partition and a 1Gb swap partition as logical volumes on the physical volume.
  • Did a base install. Noted that because I used software RAID on the whole thing, it uses LILO instead of Grub. Oh well, you can’t have everything.
  • Rebooted and the BIOS only saw one of the two disks.
  • Fiddled with the disk sled, rebooted, and this time it saw both.
  • Evidently the first boot without the second disk caused the raid to degrade, so re-added the disk mdadm /dev/md0 --add /dev/sdb1 and now it appears to be rebuilding.

Day 2:

  • Installed smartmontools, and enabled in /etc/defaults/smartmontools. Express slight concern that /dev/sda has an exit status of 64 because of some error in the log, probably due to the late unpleasantness. Will have to figure out how to clear that.
  • Installed munin-node and munin-plugin-extras, and copied the configuration from my backup from the last time
  • Installed openssh-server (unselect xauth which gets added automatically because it drags in a ton of X11 libraries). Copied /etc/ssh/sshd_config and /root/.ssh directories from backup.

Day 3:

  • Installed xen-utils. Holy shit that dragged in a lot of dependencies, and it said it had to “reinstall” 200+ packages for some damn reason. But then it gave an error, and when it came back it didn’t have to reinstall them after all. Very odd.
  • Didn’t see any xen in /etc/lilo.conf, so installed linux-image-2.6-xen-amd64. (Had originally thought that installing xen-utils would do that, I thought it did last time.)
  • Lilo complains that /vmlinuz is too big. According to the docs, lilo and xen don’t play together well, and grub has trouble with /dev/md0 software raid. I think I may have to go back to the drawing board, either re-installing the raid card, or going back to the primary boot partition and putting the software raid on the rest of the disk. Or maybe I can figure out how to get grub working. Once again I’m reminded of “Three Dead Trolls In a Baggie” singing “yeah, but I’ve got a girl friend and things to get done”.

Day 4:

  • Reinstalled the Adaptec RAID card, and set up a hardware RAID-1
  • Partitioned the “drive” with three partitions, one 4G ext3 for /, one 1G swap, and the rest as a physical volume for a lvm.
  • Installed on /, and when it went to reboot it got to “shutting down md0” and then hung. Will have to check that again. But at least it installed Grub instead of LILO.
  • After it booted, tried the “reboot” command and it worked! Yay!
  • Installed smartmontools, but discovered (once again) that it doesn’t work with the raid controller, so uninstalled it. I need to find if there is some other way to monitor the raid controller. I think I tried the dpt_i20 thing before and it didn’t work.

Day 5:

  • Installed sshd, copied the configuration from the backup to only allow public key logins. (Bite it, password guessers)
  • Installed munin-node
  • Installed linux-image-2.6-xen-amd64 and xen-hypervisor-3.2-1-amd64
  • Rebooted and the damn thing spewed tons of errors and hung. Tried to reboot with the old kernel (that worked before) and I got the same errors. I guess it’s time to give up on that hardware RAID again.

Day 6

  • Ran the disk “verify” tool in the raid card, and it didn’t find any errors.
  • Anything I tried to boot the system (the original kernel that worked before, single user mode) still failed in aacraid.
  • Ripped out the raid card again, and installed with /, /boot, /var and swap as primary partitions, and the rest of the space on both drives as a software RAID-1 used as a physical volume for LVM.
  • Install openssh-server (and unselect xauth). Copy /etc/ssh/sshd_config and /root/.ssh from backup.
  • Install smartmontools and enable it in /etc/default/smartmontoolsctl.
  • Install munin-node.
  • Rebooted to make sure everything starts correctly.
  • Installed linux-image-2.6-xen-amd64 and xen-hypervisor-3.2-1-amd64
  • Reboot again.
  • Ok, it booted, but “xm list” isn’t up.
  • Manually start xend and “xm list” is working.
  • Rebooted, and this time “xm list” is working.
  • Started to create the lvm logical volumes for the domUs

Day 7:

  • Discovered that when I backed up the last nearly successful domU, I forgot to back up the boot partition, so I’m on my own for the grub configuration.
  • Untarred my backups of the “xen2” and “xen3” domUs. Got a bunch of kernel messages about kjournald being blocked for more than X number of seconds while that was going on – I assume that’s because I was running up load averages in 7 and 8 range in the dom0, which is probably not a normal thing. I hope that just because things weren’t written to the journal immediately that doesn’t mean they were written wrong, only that I might have been in danger if things had died in the middle.
  • Installed rsync so I can restore my backup of the “xen1” domU.
  • Installed vim and removed vim-tiny
  • Restored backup with rsync --delete -aSurvx --numeric-ids /mnt/usb0/xen1/Sun/ /mnt/xen1/
  • Copy the amd64 kernel modules to the domU’s /lib/modules. cp -rp /lib/modules/2.6.26-2-xen-amd64 /mnt/xen1/lib/modules Must remember to exclude /lib/modules when I do any final rsyncing from the live domUs.
  • DAMMIT! It appears that I made /var too small again. Once it saves /var/lib/xen/save in it, the file system is full. Need to move things around again.
  • Booted into rescue mode, and moved things around. Everything seems to work now.
  • Try to rsync some newer backups.

Further updates as things progress.

The first cut is the deepest

That’s not how I expected or hoped for my first kayak race of the season to go.

Dan and Jim told me about this little race in Canandaigua this weekend. It wasn’t on the NYMCRA calendar and it was very loosely organized. The web site barely mentioned the flat water part of the race, and I was getting contradictory information on where it was supposed to start. But heck, it was only 10k and I’d been itching to see how all the work I’d done this spring was preparing me.

On race day, Dan and Paul D decided not to race for various reasons. Dan still went with me, but he was going to do a downriver paddle on the part of the river below the slalom course where the flat water course ended. So basically that left Jim, me, and Julia as the class of the race. There were a couple of C2s and a C1 that looked like they knew what they were doing, but I knew we were going to be faster than canoes. So my goal was to stay with Jim as long as I could, and not blow up and keep ahead of Julia.

The start was pretty dicey – about 20 metres after the start there was a two tunnel bridge, and a couple of idiots in a raft were blocking one of the two tunnels, so we had to sprint for it. Julia, being much lighter than me, actually got there ahead of me, but afterwards I buried myself to try to pass her and catch Jim. Also there was a submerged log very early on that my rudder hit for fortunately it didn’t twist it like logs sometimes do.

Things went according to plan. For the bit where I’d worked hard to catch Jim and for about a kilometer afterwards, my heart rate was up around 171, which is the highest I’ve ever seen it. But after I caught his wake and he slowed down a bit, I recovered to the mid 150s with occasional trips up to the mid 160s when obstacles in the river made me lose his wake and I’d have to fight my way back to it. Julia and everybody else were out of sight behind us and I was able to concentrate on keeping on Jim’s wake and going hard but not too hard, averaging just a bit faster than 5 minutes per kilometer. And that took us to about the 7 kilometer mark.

At about the 7 km mark the water got rougher. Not actually white water, but lots of standing waves and twists and turns and narrow parts of the river. I lost Jim’s wake because I was spending too much time bracing instead of paddling, while his boat that was made for those types of conditions just powered on through.

At about the 7.5 or 8 km mark, there was a narrow bit where my rudder hit a submerged log and I dumped. Fortunately there was a flat part of the bank there and I was able to very quickly get out, get the water out of the boat, and get paddling again. But the water was really cold and I was shivering by the time I got in.

And not 10 metres later I got a sweeper branch in the gut, and I dumped again. This time I was using a gigantic log, about 3 times the length of my boat, to try to get out when it suddenly came lose and started heading downstream. So I had to let it go by and then get onto shore. The shore wasn’t quite as conducive to getting back in, and I’d spent a bit longer in the water, so I was pretty cold. And Julia came by as I was finally sorted out and getting back in again.

This time I resolved not to rush it, but I started paddling and realized my rudder was way off straight. It wasn’t easy to tell, because my feet had gone completely numb and I wasn’t sure if I was even pushing the tiller bar. I had to crash into shore again, get out with my feet in the cold water, and straighten the rudder. As I was doing so, the first of the C2s, a couple of women that Jim and I had met paddling on Red Creek a week or so ago, came by.

I set off again, and I’m not sure how much further I got. This time another submerged log got me, but after the log was very deep water. This time the current was so strong it ripped off my aqua shoes and took my camelback. I had to swim hard to make the shore, and when I got there the bank was so steep and so overgrown I couldn’t get my boat out to get the water out. I ever tried to break out the rear bulkhead to make it easier to dump the water, but to no avail. After a while, while the other C2 and the C1 came by, I realized I was going to have to go back into the water to dump out the boat. I wasn’t relishing that. I actually sat down on the bank for a while to warm up. When I stopped shivering, I went in and dumped most of the water out of the boat. During that time a guy came by in a squirt boat, so I realized that not only had I lost, but there was no way I was going to even finish in the top 3 kayaks. But then I had the problem of how to get back into the boat. I pushed and pulled the boat to get it into a position where I could stand up to my waist in water, and then sort of put one leg in, brace with the paddle and get the other leg in. It worked, after a fashion, but two guys in a fat, wide fishing kayak who were yelling at each other came by while I was doing so. At least I was able to pass them.

The finish didn’t come fast enough to me, but when the water wasn’t too wavy I managed to put on a bit of speed. At the finish, Dan and Jim and Julia took charge of my boat and paddle, and one of the volunteers gave me a t-shirt so I had something dry to change into. As I mentioned I didn’t have any foot protection at this point, so I kind of limped back to my car, and Jim and Julia brought me a hot dog and a bottle of water while I warmed up.

Jim and Dan went off on another down-river paddle, and I was supposed to pick them up in half an hour or so. So I was sitting there, taking pictures of all my scratches for my Facebook page when I suddenly realized that while all I could see where superficial scratches, there was an actual “puddle” (or a conglomeration of drops, if you want to be less melodramatic, and I don’t) of blood sitting there on the sill of the car door where I’d had my feet. I look at my left foot, no problems, look at my right and just about pass out from the sight of this huge flap of flesh there. I hobbled over to the first aid ambulance, and they say “you need stitches and a tetanus shot”. Before I could say “but my car is unlocked, my wallet is in it, and somebody has to pick up Dan and Jim”, they had me in the gurney and on my way to the hospital. So now I’m trying to figure out how to contact Julia to lock up my car, secure my kayak, and go pick up Jim and Dan, and also let Vicki know what’s happening. So I’m texting Vicki like mad, but I don’t have Julia’s cell phone number. I texted Stephen, since he works with her and probably knows her number, but he’s out of town and he never got back to me. I asked Vicki to post something on Julia’s husband’s facebook wall, because that’s the only way I know to get in touch with them.

As I was being admitted into the hospital, Dan phones me up to ask where the hell I was. I explained the situation, and he and Jim said they’d take care of things for me. And sure enough, while I was getting stitched up Jim and Julia come into my room and Julia starts taking pictures. Jim suggested that Julia send one of the more gruesome pictures to Vicki. But not only had they come to visit me, they’d also brought my car and my kayak and picked up all my stuff.

Obviously I’m home now. I’ve got 5 stitches in my foot, and I’m cautioned not to wear shoes very much for the next week or so. I’m sore and tired, and the numbness is starting to wear off on the cut. And I’ve got to figure out how to get Julia to send me copies of those pictures. But I did have 7/10ths of a great first race.

Programming Tests: Useless waste of time, or massive insult?

I just had to take another one of those stupid “BrainBench” programming tests for a job I’m applying for. (I’m not planning on leaving the place I’m working now, but as a contractor you always have to be ready to jump ship.) These tests are a complete and utter crock of shit. They don’t test if you can program, all they test is if you’ve memorized every obscure and complicated part of the literally millions of lines of Java API documentation out there. A typical question will show you four snippets of code, and ask you which is the correct one. And they’re not localized down into one part of the API that might be relevant to a particular job, either.

One of the questions was about how to set up a cookie handler on a persistent URLConnection. Another was on how to set the line width in a Graphics2D line. And another was on setting up a pipe between a sender and receiver. You know, if I’d needed to do any of those things in my 12+ years of programming in Java, I might have bothered to memorize that page of the API documentation. But I haven’t, so I haven’t. Instead, I’ve wasted my time learning useless things like how to write code so that when you or somebody else comes to add some functionality a few years from now, they can figure out why you did it the way you did it, and can add their stuff without breaking what you’ve got there. And how to debug an obscure exception that only happens after the program has been running for 45 days. And how to structure a program so that it’s fast, reliable, does what it’s supposed to do and doesn’t crash. You know, frivolous stuff like that.

Evidently I’m not alone in my hatred and disgust for these stupid tests. According to The Register, Ken Thompson, co-inventor of Unix and of the C programming language, isn’t allowed to check code in at Google because he refuses to take their stupid C language test. How stupid do you have to be to insult the man who invented the language by asking him to take some test that was probably written by some 24 year old “language lawyer” straight out of school who has memorized every obscure part of the language without being able to use any of it well?

So I’m guessing that I probably did horribly on that test, and that I’m probably not going to get an interview there, and that’s perfectly fine with me because I don’t want to work with people who were chosen because they’re good at memorizing language documents. They probably write horrible code, but think they’re great because they don’t have to stop and think and look things up.

Maybe VPS or cloud *is* the way to go, after all.

More problems setting up the new colo box:

  • I shut down with all three domUs running, and when it came back up, xend wasn’t running so they wouldn’t start up again. Further investigation showed that I hadn’t made my root partition big enough to handle when it saved the current xen state to /var/lib/xend. So I made a new lvm partition for /var/lib and mounted that instead. So far, so good.
  • While I was investigating this, I noticed my software raid was running in degraded state, because it had lost /dev/sda3. I re-added it and it started to rebuild it.
  • While it was rebuilding, I noticed that xen1, my first domU wasn’t running any more. When I tried to recreate it, it told me I couldn’t recreate it because its main disk, /dev/xen-space/xen1-space, was mounted in a guest domain. Oh oh.
  • Then I got a kernel panic. Double oh oh.
  • I rebooted, and tried to rebuild the raid without xend running, but I got another kernel panic.

This is supposed to be easy and fun, right?

I’m currently booted with the rescue disk, and I’m trying to rebuild the raid again. If that dies, it might be time to cut my losses.

…but the greatest of these is love.

Our church is having a series of discussions called “Reconciling in Christ”. Basically the higher ups said “we’re going to leave it up to each congregation whether to accept gay people in long term committed relationships into the church, whether to celebrate their marriages/commitment ceremonies, and whether to allow them to be called to the pulpit”. Our church seems to be ok with the first and maybe the second, the the third part is what this is all about – we’re having bible study to discuss the meaning of various bible passages and the larger context of them which is so conveniently ignored by certain ignoramuses holding up signs on street corners or speaking at political gatherings.

Note that we specifically spell out “long term committed relationships” – we don’t want bath house cruisers or airport washroom lurkers or singles bar one night stand artists of either gender or orientation. That doesn’t mean that you have to be in a relationship right now, it means that when you do get into a relationship, you’re doing so with the hope or intention that if you’re compatible, it will last more than one night.

A lot of this discussion seems to bog down on people who can’t actually put their finger on what is wrong with a gay person finding love, but they feel in their guts that there is something wrong with it. Somebody in today’s session said “I welcome any gay person or gay couple into our church, but I don’t want them standing in the pulpit”. I wanted to ask them what they expected that person to do if they were in the pulpit? Make constant references to how great gay sex is? Hit on their sons and daughters? Show up in an outrageous Pride Day costume? Because I don’t see heterosexual people in the pulpit doing any of those things, and I don’t think anybody would tolerate it if they did. (What was it Dan Savage said? That Halloween has become like the hetero equivalent of Pride Day? Yeah, we wouldn’t tolerate our hetero pastors showing up dressed as the naughty school girl or the slutty nurse either.) On the other hand, maybe they’ll just be up there and be speaking of how great it is to have a partner they love? Would that be awful?

But the problem is that I don’t the great and overwhelming love of Christ that other members of the Church profess. To me, he was a possibly-divine person who preached an abiding message of peace and love. What I have a great and overwhelming love of is love. I wish everybody could spend every waking hour thinking how fortunate they are to love their partner the way I love mine, and the way I believe she loves me. I wish they could know that happiness that says that even when we’re apart, even when we’re doing our own things, even when we fall short of what we should be doing, even when we fight, we know that person is there and loves us back as much as we love them. And that is the greatest gift, whether it came from God or from evolution. And I’m not going to say “your love is wrong” because of the way your genitalia is arranged. If you love somebody, and they love you back, then you are well and truly blessed.