How to crash the data center with one word

A single word on a simple button does not mean what an admin thinks it means during what should be a routine weekend job

Techies and users often accuse each other of speaking different languages, but truth be told, we in IT don't always understand one another either. Take the case of the tech team who decided a simple one-word sign in the data center would mean the exact opposite of all expectations.

Many years ago, I was hired as a junior sys admin (aka systems programmer) for a mainframe data center. In all, our department had 12 people at the site.

At first, I worked during business hours only, with the rest of the staff present. But one weekend, I needed to go to the office to take care of some tasks in the data center. I chose to go during off hours when the update I had planned wouldn’t disrupt the rest of the employees.

Our mainframe data center was small and unattended on weekends. I had a key to the building, let myself in, and made my way to the data center.

A logical conclusion

The door was closed and locked, which of course wasn’t surprising as this was the weekend. But standing outside the door, it dawned on me that I had always entered the room when it was open. I didn’t have a key or know how to get in. My supervisors must have assumed that I knew how when I’d mentioned my plans to come by over the weekend -- I was never given instructions, nor did I think to ask.

As I paused outside of the closed door, I stared at a big, red button labeled "Open" right next to the entrance. I’d seen it before and assumed it would open the door. Logically, I pushed the button.

A calamitous outcome

Instantly, a dreaded silence descended -- the sound of a data center that has gone dead. The Open button must have shut off all power to the data center! It certainly hadn’t opened the door.

I called the site manager to find out how to get into the data center as fast as possible. She told me where to find the key to the data center door. I located it, opened the door, and confirmed my suspicion: The entire data center -- including power, cooling, lights, mainframes, disk drives, communications controllers, everything -- had gone down hard.

I was able to restart the systems, but not the A/C. I had to call the site administrator, who was able to get the A/C going. The good news was that the system was largely quiet on weekends, so there were no critical updates in progress or batch jobs running. Nothing was actually damaged.

Long into the evening, all the systems were finally up again for business the next day. No harm was done in the end and the users were none the wiser. I endured a little good-natured teasing from the rest of the staff, of course.

To this day I marvel at the lunacy of putting a big, red, completely unprotected button next to a door, labeled as Open but in actuality meaning “Open all power circuits in case of emergency only.”

The label never changed, but our crew put a plastic box over it so you had to flip open the box before you could push the button. Even with a small staff, you can never assume that everyone knows what a sign like that means. Clear communication is a necessity.

Copyright © 2014 IDG Communications, Inc.