How Not To Test A New System: Push A Button And Wait To See What Happens

Who, me? Welcome once again, valued reader, to Who, Me? – The Register's weekly confessional column in which readers recount their tales of derring-do that derring-didn't.

This week meet a reader we'll Regomize as "Doug" who found himself in something of a hole when, as an ops manager for a food supply company, he oversaw a massive upgrade of IBM AS/400 F-Series kit. Six racks worth of it, including all the trimmings – mirrored disk protection as well as tape backups.

You can, after all, never have too many backups.

Obviously such a huge upgrade was not going to be done entirely by the customer's techies, so IBM supplied several engineers to do the serious voodoo parts, as well as a mobile disaster recovery vehicle. The plan was in four stages: build the new system in its dedicated space; transfer everything to the IBM van in the carpark; then transfer it all to the new system. All with appropriate testing of course. No chances were being taken.

The first two stages went well, and the van kicked into gear to provide the required services. Then the IBMers with their "eyes only" red books of engineering secrets set about mirroring to the new kit.

The third stage saw everything mirrored to the new system and the OS up and running on the new kit. All going well so far.

• Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else
• Just follow the instructions … no wait, not that instruction to lock everyone out of everything
• Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?
• The boss worked in a fishbowl, so office tricks were a treat

This is the point where Doug intervened. Chatting amiably to an IBM engineer named "Bob", Doug casually mentioned that the mirroring hadn't been tested yet and – with barely a thought to the consequences – hit a power button on the rack, switching off the disk array and tape backups. He had, he tells us, a grin on his face.

Bob did not have a grin on his face. Bob was horror-struck. Because of course the mirroring hadn't been tested yet – and just shutting it off without warning is not the IBM-approved way to test it.

Gathering his wits about him, Bob immediately ran diagnostics on the parts of the system that Doug had not just casually killed, and found that all was OK.

Doug, relieved, reached out to hit the power button again, only to find himself physically restrained by an exasperated Bob. Suddenly turning the drives off is not in the manual, nor is suddenly turning them on again. And if it isn't in the manual it ought not to happen.

While Bob consulted with more senior engineers, Doug had to explain the situation to his own head of IT, as well as the logistics companies that were going to be relying on this kit once it was up and running. And of course to the board of directors.

In the end it was all smiles. The upgrade was successful, productivity boosted and Doug, forgiven, was promoted to head of IT.

But he learned an important lesson that day: never turn anything off if you don't know how to turn it back on.

Have you ever learned an important lesson the hard way? Found out that sometimes it's better not knowing what that mysterious red button does? Tell us all about it in an email to Who, Me? and we'll make you (anonymously) famous. ®