Boss Broke Servers With A Careless Bit Of Keyboarding, Leaving Techies To Sort It Out Late On A Sunday

On-Call Give thanks, dear reader – the weekend is upon us and it's a lovely long one for readers in the US of A. But that's not stopping us from bringing you another instalment of On-Call, The Register's weekly reader-contributed tale of the thankless chores IT pros are asked to perform.

This week, meet "Dennis", who in the early 2000s worked for a managed service provider that was in the midst of a datacenter migration.

"Our racks were in a literal cupboard and a newer room was being built next door," Dennis explained.

While the build progressed, Dennis and his colleagues started moving odd bits of equipment around and tidying things up.

"All was going well," Dennis reminisced.

Until it wasn't.

"On Sunday evening we noticed that things were running slow but didn't think much of it – we just assumed an automated process was taking longer than usual and things would calm down."

Things did not calm down. Instead, they heated up to the point at which everything crashed.

"Late that night everything died. Our firewalls, servers, the VSA platform we used to manage clients' kit, and an on-prem email server."

• IT manager's 'think outside the box' edict was, for once, not (only) a revolting cliché
• Go ahead, be rude. You don't know it now, but it will cost you $350,000
• Hot, sweaty builders hosed a server – literally – leaving support with an all-night RAID repair job
• No, I will not pay the bill. Why? Because we pay you to fix things, not break them

Dennis's colleagues dragged themselves awake and into the datacenter, where they found the air conditioning had shut down and server fans were roaring.

The culprit was a keyboard.

A keyboard?

"The boss had been in earlier that day to do more work on the rack, and had left a keyboard on the floor standing on end, leaning against the fans," Dennis recounted. Combined with pressure from a door, the keyboard had enough force to do some damage.

Dennis found some upside in the meltdown.

"We intended to replace a pair of servers with newer models when the new server room was ready," he explained. Thankfully those servers were already in the building. Not so thankfully, the keyboard incident meant Dennis got to spend Sunday night and most of Monday rebuilding everything.

Dennis also got to respond to some customer complaints about the outage with an explanation that started "The boss came in on Sunday and left a keyboard lying around …"

Has seemingly innocuous equipment caused you grief? If so, click here to send On-Call an email and we may thank you by using your story in a future column. ®