I’ve spent the last couple of months doing an all-singing, all-dancing automatic upgrader for our customers sites. This process is designed to be totally hands-off – you stick the DVD in the drive and type “upgrade”, and at the end of the theatre day it will convert the main “cms” computer (one per site) and all the “cp” computers (one per projector) from Redhat 7.3 to CentOS 3.4, upgrade from version 3.3 to 3.6 of our software, and magically preserve all your settings and configuration. You should come in the next day to find everything ready for the day’s schedule.
For the very first one at a customer site, though, they sent out a technician to babysit it. Unfortunately they send a techician who’d never done or witnessed one of our many test upgrades in-house.
You probably guessed what happened – she saw the cms come up, didn’t realize that the cps start after the cms is done, and rebooted the cms at the worst possible time – right when all 18 cps were attempting PXE (network) boots and expecting the cms to be there to send them what they needed. And the cms doesn’t start dhcpd by default, so the cps have had nothing to talk to all night. And of course everybody is screaming for me when I got in!
“screaming for” beats “screaming at” any day.