Step away from the button! 6 touchy tech disasters

It's all fun and games until you bring down the data center with a stray brush of the hand or flick of the finger

Gunnar Pippel

We can't keep our hands off tech and gizmos. We give in to the urge to see what happens if a button is pushed or naively believe it couldn't hurt anything. Then chaos breaks loose and IT spends hours repairing the damage.

The InfoWorld Off the Record blog is where IT pros share real-life tales from the job. In this collection, rookies and veterans alike are caught up when a flip of the switch brings the harmonious hum of work to a sudden halt.

Share your experience about managing IT, developing apps, supporting users, or a humbling moment. Send your story to offtherecord@infoworld.com. If we publish it we'll keep you anonymous -- and send you a $50 American Express gift cheque.

The circuit breaker vs. the network admin
Robert Acocella

The circuit breaker vs. the network admin

At budget time, a senior network admin shows off the data center to the managers, especially two newish servers. The admin brags about the uptime and waves away questions about a UPS, saying there's no need. To prove his point, he touches the server's dedicated circuit breaker -- and 500 users instantly lose their server connection. The second server proves just as cooperative, severing the connection for the rest of the employees with merely a touch of the circuit breaker.

Somehow, the admin does keeps his job, if not his pride. Bonus: Management allots more money for server infrastructure.

Don't get caught with your pants down
Katja Govorushchenko

Don't get caught with your pants down

The good news: You catch a mistake before damage is done. The bad news: You're not out of the danger zone.

A junior techie goes to the server room to turn off a noncritical machine. The techie finds it and pushes the button -- then realizes he had the wrong server, one housing the files in use by 600-plus employees. Letting go of the button would mean chaos.

With no cell service and nobody within earshot, his only hope is a phone standing 10 feet away. A desperate search yields an unusual solution: The techie removes his pants, throws them around the phone, and finally pulls the phone in. Help eventually arrives -- along with several other gawkers who don't forget the sight soon.

Idle hands are the enemy of IT
Andrew Lewis

Idle hands are the enemy of IT

The data center operator only wants to expedite tasks, ease workflows, and clean up nagging details. But one day the zeal for improvement goes a little too far.

The efficiency bug bites back when the operator notices a loose ring on a piece of glass -- that happens to cover an emergency power-off button linked to the data center. A tiny turn of the ring sets off a chain of events, including a systems shutdown and a blackout. No long-term damage is done, but the operator learns to investigate thoroughly before making "improvements" -- and when to keep his hands to himself.

A clip unleashes a cavalcade
Ron Chapple Stock

A clip unleashes a cavalcade

Sometimes, an assuming desktop item can unleash havoc at the office.

Picture the classic computer room: three mainframes bearing multiple attached tape drives, a row of disc packs the size of dishwashers, three line printers, one high-speed laser printer -- and a massive Halon fire suppression system. The boss and the operators can't agree on how to handle a shutdown in case of fire, with the boss insisting the operators stay behind to handle the mainframes. To settle the matter, they gather for a test run.

Before they can start, a stray paperclip drops into the control box, creates a short circuit, and triggers the Halon. Guess who's the first one out the door? And the operators prove their point handily.

Power players pull a power play
livechina

Power players pull a power play

Here's a simple rule for admins: Do not let the execs touch the servers. Repeat: Do not let the execs touch the servers.

Otherwise, you may suffer the same fate as two tech departments in the midst of an arduous data center consolidation after a merger, complete with layers of paperwork and permissions. You wouldn't have known it, though, on the day workers can't get email and some remote sites can't connect. The staff eventually traces the problem to server failures and wonders why it all happened at once.

Coincidence? No. The VPs had visited the acquired company earlier that day and turned off "unused" servers they determined had no impact on productions systems.

Repeat: Do not let the execs touch the servers.

An inside threat
Thinkstock (Thinkstock)

An inside threat

Stupid users, clueless execs, bumbling bosses -- you expect trouble from them. But what happens when a colleague within IT is responsible for that fateful flick of the wrist? In this case, a near disaster.

The tech pro should know better when tasked with determining which servers are to be decommissioned in a large, bustling data center. Be it a brain fart, bad judgement, or simple ignorance, he hones in on a critical management server -- then unplugs it, takes it to his desk, and reformats the hard drive.

This major mistake unleashes a tidal wave of problems related to the firewalls, backups, and that database. Unsurprisingly, the techie is transferred before he could strike again.

Have an IT story to share?
iStockphoto

Have an IT story to share?

Write up your own crazy-but-true tale about managing IT, developing apps, supporting users, a humbling moment, or a time when something went very right.

Send your submission to offtherecord@infoworld.com. If we publish your story -- anonymously, of course -- you'll receive a $50 American Express gift cheque.