Datacenter Migration Plan Missed One Vital Detail: The Leaky Roof

On Call If it's Friday it must be time for another episode of On Call, The Register's weekly column celebrating readers' escapes from nasty scrapes.

This week, meet "Danny," who in 2016 started a new job with an automotive company and was immediately given the task of renovating the organization's datacenter.

"The project included replacing four power distribution units and moving all power and data cabling from under the floor to new overhead cable trays," Danny explained. Doing so meant eliminating the overhead drop ceiling to make space for the cable trays, plus replacing ancient fluorescent lighting with new LED kit.

All that work necessitated an expansion of the fire suppression system and raising the dry sprinkler heads from about 18" above the racks to the ceiling itself.

Just to make sure the job was especially difficult, this had to be done while the datacenter kept working.

Come the day and the racks were covered in protective sheets so the ceiling could come down without squashing servers or clogging fans with dust.

But things quickly went pear-shaped when that demolition of the ceiling tiles revealed a lattice of cinder blocks above the datacenter, rather than the expected pleasingly solid slab of concrete.

The situation quickly worsened when the site facilities director arrived and demanded an explanation for the mess.

The director accepted Danny's explanation, but then insisted the recently destroyed ceiling be restored.

"It's the only thing protecting the racks from the ceiling leaks if the upper floors flood," Danny was told.

"Ceiling leaks?" Danny asked, incredulous, before being told that several times in preceding decades sprinkler systems had accidentally triggered. The resulting torrents of water had found several holes and cracks in the cinderblock ceiling, raining water onto the racks of equipment and damaging devices.

The site facilities director explained that water sensors had been installed to detect such floods … but the organization had come to assume that some water would pool above the ceiling tiles which would provide enough of a barrier to stop the datacenter getting serious soaking.

This news "caused a sudden immediate stop to the project as we searched for and found contractors who specialized in sealing any penetrations in an overhead ceiling to make it watertight," Danny told The Register.

Three months and several hundred thousand dollars later, the roof was sealed, procedures had been written to cope with leaks, and Danny was able to finish the job.

"The rest of the project completed without significant incident and from then forward, any new IT leadership visitor was brought through to see the renovated datacenter and allowed to see the uniquely sealed cinderblock ceiling," Danny concluded.

Have your renovations ever been mugged by reality? If so, let us know with an email to On Call and your story could feature here on a future Friday. ®

RECENT NEWS

Google Leverages AI To Automatically Lock Phones During Theft

Amid increasing incidents of mobile phone thefts, Google has launched an AI-based feature that automatically locks the s... Read more

Microsofts Emissions Surge Nearly 30% Amid AI Demand Growth

Microsoft has reported a nearly 30% increase in its emissions from 2020 to 2023, underscoring the challenges the tech gi... Read more

Impact Of AWS Leadership Change On The Global AI Race

The recent leadership transition at Amazon Web Services (AWS), with Adam Selipsky stepping down and Matt Garman taking t... Read more

The Global Impact Of App Stores On Technology And Economy

Since Apple launched its App Store in 2008, app stores have become a central feature of the digital landscape, reshaping... Read more

Alibaba's Cloud Investment Strategy: Fuelling AI Innovation And Growth

Alibaba Group's cloud business, Alibaba Cloud, has emerged as a powerhouse in the tech industry, spearheading innovation... Read more

Elon Musk Takes On Government 'Censorship': A Clash Of Titans In The Digital Arena

Elon Musk's recent endeavors to challenge government-led content takedowns mark a significant development in the ongoing... Read more