Yesterday, while trying to fix the problems with my mailman mailing list, I decided to rebuild the archives on the mailing list that was giving me problems. But I got the syntax of the “for i in *; do ... done” command and instead of running mailman’s arch command with carefully snipped out parts, it instead ran it with the whole archive. And that meant that arch quickly chewed up all the swap space available. I became unable to kill it, and quickly lost connections to both the domU in question, and the dom0. I couldn’t even ssh back into the dom0.
Not being clear of thought, I emailed the colo company asking if they could power cycle my box. 5 minutes later I realized that all my out-going mail goes through the colo box so it wasn’t going anywhere, and so I phoned them the request instead. They power cycled, I got control of my colo box again, and I got the list fixed up and the archives rebuilt.
But I noticed that this email to the colo company was still sitting in the outbound queue on the colo box, hours later. I didn’t think anything about it, until about 10 minutes ago I got a response to it, dated 10 minutes ago, saying “ok, I’ll power cycle it now”. I immediately fired back a “NOOOOOOOOOOO!!” email, but of course it was too late – the box went down, and now it’s back up.
And I notice that my email to the colo company is still just sitting there, with
88AB94F0AFB 1094 Tue Dec 5 10:54:49 ptomblin@xcski.com
(connect to mail.annexa.net[66.162.186.199]: Connection timed out)
annexa@annexa.net
Something tells me that email isn’t the best way to talk to these guys.