Datacenter Migration Plan Missed One Vital Detail: The Leaky Roof

On Call If it's Friday it must be time for another episode of On Call, The Register's weekly column celebrating readers' escapes from nasty scrapes.

This week, meet "Danny," who in 2016 started a new job with an automotive company and was immediately given the task of renovating the organization's datacenter.

"The project included replacing four power distribution units and moving all power and data cabling from under the floor to new overhead cable trays," Danny explained. Doing so meant eliminating the overhead drop ceiling to make space for the cable trays, plus replacing ancient fluorescent lighting with new LED kit.

All that work necessitated an expansion of the fire suppression system and raising the dry sprinkler heads from about 18" above the racks to the ceiling itself.

Just to make sure the job was especially difficult, this had to be done while the datacenter kept working.

Come the day and the racks were covered in protective sheets so the ceiling could come down without squashing servers or clogging fans with dust.

But things quickly went pear-shaped when that demolition of the ceiling tiles revealed a lattice of cinder blocks above the datacenter, rather than the expected pleasingly solid slab of concrete.

The situation quickly worsened when the site facilities director arrived and demanded an explanation for the mess.

The director accepted Danny's explanation, but then insisted the recently destroyed ceiling be restored.

"It's the only thing protecting the racks from the ceiling leaks if the upper floors flood," Danny was told.

"Ceiling leaks?" Danny asked, incredulous, before being told that several times in preceding decades sprinkler systems had accidentally triggered. The resulting torrents of water had found several holes and cracks in the cinderblock ceiling, raining water onto the racks of equipment and damaging devices.

• Keeping printers quiet broke disk drives, thanks to very fuzzy logic
• Using the datacenter as a dining room destroyed the platters that matter
• Doctor gave patients the wrong test results due to 'printer problems'
• Terminal downgrade saves the day after a client/server heist

The site facilities director explained that water sensors had been installed to detect such floods … but the organization had come to assume that some water would pool above the ceiling tiles which would provide enough of a barrier to stop the datacenter getting serious soaking.

This news "caused a sudden immediate stop to the project as we searched for and found contractors who specialized in sealing any penetrations in an overhead ceiling to make it watertight," Danny told The Register.

Three months and several hundred thousand dollars later, the roof was sealed, procedures had been written to cope with leaks, and Danny was able to finish the job.

"The rest of the project completed without significant incident and from then forward, any new IT leadership visitor was brought through to see the renovated datacenter and allowed to see the uniquely sealed cinderblock ceiling," Danny concluded.

Have your renovations ever been mugged by reality? If so, let us know with an email to On Call and your story could feature here on a future Friday. ®