I’ve mentioned before that in order to help defray the costs of putting my stuff on a colo box, I partitioned the box in 3 Xen virtual machines, and rent two of them out. Well, yesterday one of the renters, Terry, asked for a bit of help with his Apache set up. Not knowing his root password, I mounted his hard drive in the “dom0” Xen controller, using “mount /dev/xen-space/xen2-disk /mnt” and started poking around. Well, evidently that managed to confuse ext2 because a few hours later he emailed me to say that his disk had gone “read-only”, and when he tried to reboot it didn’t come up.
Looking at my munin graphs, it appears that when he rebooted, it took down the whole box. I had to email the owner of the rack to power cycle my box, which he can do remotely. When it came back, 2 of the 3 virtual machines came up fine, but Terry’s was asking for a root password to run fsck. I shut down his virtual machine and did a fsck from within the dom0, and it found several things out of whack. But after those were fixed, I was able to restart Terry’s virtual machine.
So lesson learned. I’m not sure if things would have been happier if I’d mounted it read-only, but in the future if I need to mount one of the partitions in /dev/xen-space I’ll shut down the xen virtual machine instance first.