Why didn’t I use LVM on everything?

Due to a series of historical accidents, I have the following disk space layout:

Partition Size Use
Disk 1 – 250Gb
/dev/hda1 2Gb dom0 root
/dev/hda2 2Gb dom0 swap
/dev/hda3 Rest part of vg “xen-space”
Disk 2 – 250Gb
/dev/hdc1 2Gb formerly dom0 root – unused
/dev/hdc2 1Gb formerly dom0 swap – unused
/dev/hdc3 Rest part of vg “xen-space”
Disk 3 – 400Gb
/dev/hde1 300Gb mounted as /dev/hdb on a domU
/dev/hde2 Rest part of vg “xen-space”

The root partitions of the three domUs are all lvs on vg “xen-space”. There is over 250Gb free on the vg.

What I would like to do is clean up the second drive to get rid of the extraneous partitions, and to grow the partition on /dev/hde1 to the full disk. So what I’m thinking of doing is the following:

  1. Migrate the content of /dev/hdc3 off using “pvmove” and “vgreduce”.
  2. Delete all three partitions on /dev/hdc3 and add it back to the vg using “pvcreate /dev/hdc; vgextend xen-space /dev/hdc”.
  3. Migrate the content of /dev/hde2 off using “pvmove” and “vgreduce”.
  4. Delete the /dev/hde2 partition and increase the disk of /dev/hde1 to fill up the drive, and use resize2fs to make /dev/hde1 use the whole partition.

The problem is that I don’t know if I can do this stuff without shutting down the domUs. And for the physical partition /dev/hde1, which is mapped to the /dev/hdb on one of my domUs, I don’t know if I have to shut down the domU or just umount it within the domU and remount it afterwards.

That was relatively painless

The box is up. It was only offline for about 20 minutes, tops. Everything seems to be working. Fingers crossed.

I officially love LVM. And I wish my big binary file archive was on LVM, because it’s getting nearly full and I want to expand it. Of course, any changes in it will probably require shutting down because of the way the file system is mounted through Xen.

Moment of truth

Tomorrow morning, I go out to the colo box and replace the existing one with my spiffy “new” Yellow Menace. I’ve tested the hell out of this one and it can handle all three domUs and the dom0 all doing dd’s from /dev/zero to the hard disk over and over again, which will be a nice change from the existing one freezing up throwing ext3 errors whenever I’m doing something disk intensive.

My plan is to pull out the old one, move the disks to the new one, boot it up, make sure I can log into the dom0 from home, and then go home to tinker with it. If things go as I expect, the services that live on here, like this blog, my mailing lists, my photo gallery, the Rochester Flying Club web site and others should only be off the air for an hour or so.

Then I’ll test the hell out of the old box when I get it home to see if I can reproduce the problem, and then see if I can fix the problem somehow, maybe with a new IDE cable. Then I’ll know how to advertise it on eBay – either as a working box, or a box with a suspect IDE controller but a probably fine SCSI controller.

My new colo box

It arrived last night. So of course I had to spend the whole damn night setting it up, in spite of the fact that the colo guys won’t be there to let me in for a few days.

Isn’t it pretty? As you can see, it’s a former Google Search Appliance, but without the Google software. Google evidently didn’t want anybody messing with it, and they even included a modem that you were supposed to connect to it in the case of problems so they could remotely diagnose and/or fix it. Which leads to problem number 1.

See that faceplate offset from the box in this picture? When it first came, that faceplate obscured access to the disk caddies, the floppy and the CD-ROM. Which would cause some difficulties installing new hard drives and booting from a CD to install the OS. The screws holding the faceplate on had had their (whatever you call the part the screwdriver goes in) rounded out so you couldn’t use a screwdriver to remove them. That necessitated solution number 1.

We ran out to the hardware store and bought a Dremel tool, and used that to cut grooves in the top of the screws so I could get them off. Once that was done, it was remarkably easy to get everything up and running. The drives are in caddies that slide out the front, and each drive is on a separate IDE controller – it appears that /dev/hda is the first disk, /dev/hdc is the second, /dev/hdd is the CD-ROM, /dev/hdc is the third disk. The only catches in the installation are:

  • There is a BIOS password that I don’t know – there is a jumper that says it will clear the CMOS, but I haven’t had the nerve to use it in case it clears something else that I don’t want cleared.
  • There are two ethernet jacks, and the one recognized by the Linux controller as eth0 is not the one you’d expect. Also, if you set the jumper to disable the ethernet controller, both seem to stop working. Either that, or the other one is a different type of ethernet controller that the Xen kernel can’t deal with.

I’d been using my Windows box as a sacrificial test machine with a couple of scratch disks to emulate what I have at the colo facility to play around. For the last couple of weeks, I’d been convinced that this was going to mean that I was going to have to take the box home to transfer the disks and install the latest Xen kernel, because I couldn’t get it to work consistently. But the new box has two things going for it – it has three disk slots, and the current colo boxes are using LVM. Experimenting with the new box showed that I could install a basic Debian Sarge on it, upgrade it to Debian Etch, use the Xen packages in Debian Etch to get up and running, and then slap the disks from my test machine in, and even though they’re on different devices than they were on the test machine, LVM automagically recognized them and I was able to mount the domU partitions exactly the same as I had in the test machine. From there, it was a simple matter to copy the new /lib/modules/2.6.18-3-xen-686 to the partitions, make a few small changes to the domU config files, and start them up.

This is great – it means that the way things are now, I can take the new box over to the colo, swap the hard drives over, and boot it up and it will be in a state that I can do the rest of the setup from home, with no data loss and only a few hours of downtime. That is currently scheduled for next Wednesday.