Nobody Does DR Tests To Survive Lightning Striking Twice

On Call Lightning may never strike twice, but each and every Friday The Register runs another edition of On Call, our reader-contributed tales of tech support gigs that did not spark joy.

This week, meet a reader we'll Regomize as "Karl" who once worked in the services limb of an enormous and venerable three-letter corporation's global services operation.

"We supported the mainframes and datacenter operations for a large international hotel and resort conglomerate" Karl told On Call.

Nothing quite enhances the senses like a dark, quiet and smoky datacenter

Karl and his colleagues were required to clock 45 hours a week on their timesheets, but he rated the gig "fairly cushy" and mostly involving "server crashes and failed disks."

But the night a thunderstorm rolled past the hotel conglomerate's main datacenter was not cushy.

"Phones and pagers started going off around the world," Karl recalled.

Upon arrival at said facility, Karl found a tree split in half near its exit. That was a graphic clue about what had happened: lightning had struck, backup power had failed, and acrid smoke had everyone spooked. The main casualty was a power conditioner near the exit door, and its demise caused a power surge that "jumped every safeguard, and took out nearly every Ethernet port in the building too."

Under emergency lighting – which survived the lightning strike – Karl observed that "Nothing quite enhances the senses like a dark, quiet and smoky datacenter."

In that spooky atmosphere, Karl and his colleagues "spent 12 hours restoring service, swapping parts from spares, and parts from the local depots, alternating shifts to keep people fresh." Happily, the mainframes came up, and seemed no worse for wear. After a long slog – performed under the pressure of the hotel chain's desire for mission critical apps to resume operations – the end of the job was within sight.

• Quirky QWERTY killed a password in Paris
• False negative stretched routine software installation into four days of frustration
• That old box of tech junk you should probably throw out saves a warehouse
• Cunningly camouflaged cable routed around WAN-sized hole in project budget

We opened this On Call with the adage that lightning never strikes twice.

As he prepared to finish the restore job, Karl learned that's not true.

"A second lightning strike sent a second power surge coursing through the building, throwing all power out the window, and we literally just sighed, and flipped our playbooks back to page one, assessing the situation from scratch."

"The same fire crew showed up, the same techs were called, the same parts from stores were pulled out, the same careful fingers-crossed power cycle of the mainframes occurred and amazingly, they all came up without a problem."

After more than 24 hours of disaster recovery, Karl could pack up and go home.

"The hotel chain lost a little revenue, and the big global services firm got a fat check," he recalled.

But Karl and his pals got nothing.

"Not a pat on the back, a pint at the pub, or any overtime pay."

Which was why, not long after this incident, sysadmins sued for not paying overtime.

"They lost, we won. But didn't really win either," he observed, before adding: "The moral of the story is 'Nothing stops lightning, and no one wins but the lawyers'."

Have you ever worked on a job that turned into double trouble? Or not been properly paid for your troubles? Click here to send On Call an email detailing your experiences and we may give it a run here on a future Friday. ®