Hot, Sweaty Builders Hosed A Server – Literally – Leaving Support With An All-night RAID Repair Job

On Call As two precious days of free time loom, The Register offers a two-minute taste of the energy-draining drama that makes weekends so important in another instalment of On-Call, our weekly reader-contributed tale of the messes IT people are asked to address.

This week, meet "Oswald" who sent a story of pulling an all-nighter to restore an email server in around 2003.

Oswald had not planned to work all night. Indeed, the office in which he toiled was no place for it – part of the building was being reconfigured to increase the size of the server room.

"The project involved building the new server room first, moving the kit in, then demolishing the old one so the main building work could begin," Oswald explained to On-Call.

While all the necessary wiring was arranged, the main aircon was out of action. A temporary unit was placed in the new server room and a substantial hole was cut in the wall so its hot air could be vented to the rest of the construction site.

This arrangement worked, and the email server Oswald tended kept humming. Until it didn't.

"I'd built the email server myself and it had always been reliable, so I went down to the server room to have a look, avoiding all the rubble and dust as I walked through the building site," Oswald explained.

He then pieced together the problem.

"It was a hot summer, and the builders had understandably started to tire of having hot air con exhaust blown over them as they worked. So they figured a solution: add more ducting – around ten meters worth – and redirect the hose out of a window."

• Switch survives three hours of beer spray, fails after twelve
• No, I will not pay the bill. Why? Because we pay you to fix things, not break them
• To make this computer work, users had to press a button. Why didn't it work? Guess
• No, working in IT does not mean you can fix anything with a soldering iron

That sounded like a splendid plan – who wants to encounter hot, sweaty, builders?

For Oswald, that terrifying prospect may actually have been preferable. As it turned out, that ducting was the reason his email server died.

"The portable air con didn't have the power to push its exhaust down that length of corrugated tubing," he explained. So lethargic hot air met construction dust and the server's air filters clogged, things heated up, and its disks degraded until email crawled to a halt.

"I proceeded to spend nearly 24 hours (including taking the server home in a taxi that evening, and back into work the next morning) coaxing the bits off the disks onto new ones." That enormous effort was needed because, while Oswald's employer had backups, the incident would have cost about 12 hours of emails.

"The company would rather be without email for a day than lose data," Oswald explained, so "In the end I got every bit off the old RAID disks and reinstalled the server early the following morning with no data loss."

The word "hero" gets used too much these days. But what the hey, The Register might just use it now.

Have builders ever botched your job? Have attempts to extend infrastructure come up short? Let us know by clicking here to send a mail to On-Call and we'll try to feature your tale here on a future Friday. ®