I noticed my backup USB drive wasn’t mounted, so my nightly backups hadn’t been happening for I don’t know how long. And when I tried to mount it, it mounted in USB “fast speed” mode, rather than “high speed”. I tried a few things, and I still couldn’t do it. So what the hell, I thought, it’s been over 6 months since my last reboot – yum has installed several new kernels, this will be a chance to use one of them.
Nice plan, but unfortunately the damn box wouldn’t boot. The first message I saw:
INIT: version 2.85 booting
mount: error while loading shared libraries: libcryptsetup.so.0:
cannot open shared object file: No such file or directory
awk: cmd. line:1: fatal: cannot open file `/proc/mounts’ for reading
(No such file or directory)
I booted with a rescue disk, and discovered that the thing it’s complaining about, libcryptsetup.so.0, is in /usr/lib. And guess what: /usr is a different partition than /, so it isn’t mounted before /etc/rc.sysinit runs. After a bit more investigation, I found that /usr/lib/libcryptsetup.so.0 was installed by a weird rpm, and that it’s used in mount and umount. I couldn’t understand why the Fedora team would be so stupid as to disobey one of the first tenants of Unix: that the system should be able to boot without /usr. mount is installed by the util-linux rpm, and I had this strange util-linux…crypt.i386.rpm. That word “crypt” in the name was evidence that it wasn’t a normal rpm. And “rpm -qi” showed that it didn’t come from Fedora. That’s when I realized that when I’d installed the yum repository for “postgrey”, I hadn’t excluded all the other packages from that repository like I should have. And this had dragged in these crappy packages.
The first thing I did was try to “work around” the problem with files in /usr/lib by copying libcryptsetup.so.0 to /lib and booting again. But that failed, complaining about another file. So I went back to the rescue disk and tried the same work around with that file, but this time it complained about yet another missing file. I realized this could be a long process.
So now I had to get rid of the bogus versions of mount and the other files that depended on this stupid crypto stuff.
Next thing I did was go to the proper Fedora upgrade site with my laptop and download the latest real version of the util-linux rpm. Then I transferred it over to the Linux box using a USB pen drive (lovely useful things, aren’t they?). The rescue boot image mounts your Fedora file system under /mnt/sysimage, so I did a “chroot /mnt/sysimage” and tried to do an “rpm -Uvh –force” on that rpm, but it didn’t work. It gave some ioctl error. At this point I really wasn’t sure what to do. I contemplated trying to combine my / and /usr partitions into one big one and making that the boot point, but I didn’t relish the work it would take to stitch everything back together.
I started downloading the Fedora Core 5 DVD image as a backup plan, but Bittorrent was telling me it would take 6 hours.
As a desparation move, I tried booting a Knoppix 4.1 boot CD I had lying around. I manually mounted the disk partitions where they belonged under /mnt/hda6, and chrooted to it, and this time “rpm -Uvh –force” actually worked. Hooray!
I booted, and had a couple of minor hickups – nfsd had taken a socket that spamd wanted, so I was getting no spam filtering until I figured out who had the socket open and restarted everything that needed to be restarted. I also had to restart a ssh tunnel on my linode to get the mail flowing correct. I think everything is working – my blog is up, mail is flowing, news is flowing, I can NFS mount my music onto my laptop, and all seems right in the world.
But I’m up, I’m running, and I’m a little less sanguine about installing new repositories into yum. Time to go to bed.