An IT Emergency During A Festive Visit To The In-laws? So Sorry, Everyone, I Need To Step Out For A While

On-Call Much of the world may be on holiday, but On-Call – The Register’s weekly tale of readers being asked to rescue tortured tech, is still hard at it.

This week, we offer a topical tale from “Bill” who spent one festive season in the company of his in-laws and had decided to bring his work laptop along because previous experience had taught him that it might be wise to do so.

Bill’s decision was prescient.

“One day while out driving around, I got The Call from my manager,” he told On-Call. “The IT staff we inherited from a recent acquisition were having problems recovering from a tape backup, and as the resident backup expert, they wanted me to help.”

Help was certainly needed: the main production server had become corrupted on Saturday, and the Friday night backup wasn’t restoring as expected.

All that was left was Sunday’s backup of corrupt data.

• Server broke because it was invisibly designed to break
• Boss installed software from behind the Iron Curtain, techies ended up Putin things back together
• Two signs in the comms cabinet said 'Do not unplug'. Guess what happened
• Go ahead, be rude. You don't know it now, but it will cost you $350,000

So while Bill’s family shopped, he sat in a coffee shop, fired up his VPN, called relevant colleagues at the acquired company, and learned that the backup system relied on a single tape drive, plus three other crucial facts.

The sole tape in that drive was replaced every Friday before staff went home, and the full backup ran on Sunday.

AS this was the silly season, the tape had not been swapped. So last weeks’ tape – with a good backup - was still in the drive.

The tape was big enough to hold multiple weeks’ worth of backups from this server.

So far, so good.

But Bill also spotted a grinch: the data protection software was set to erase the tape before each new backup.

Which meant that after the server data corrupted on the Saturday, it then wiped previous backups on the Sunday.

Bill was left to deliver the message that the previous week’s backup had been deleted, then overwritten with corrupt data.

What was to be done? Would Bill need to abandon his family, and his holiday?

“There was a very long moment of uncomfortable silence on the call,” Bill recalled, which was finally broken by his manager saying “Thank you for your help on this. You can go back to your vacation.”

So Bill did.

Upon his return, his first job was installing an eight-tape autoloader. Six months later that rig was moved to HQ and away from the acquired company’s clutches.

Who or what has an IT emergency given you an excuse to escape? Share your stories by clicking here to send your story to On-Call. Don’t be shy – the On-Call mailbag needs some new items to consider as we plan for next year’s articles. ®