When servers go silent: An admin's nightmare

Amid a routine cleanup, a tech is confronted with the deafening silence of servers gone offline

plug in connection network

As techs, we hope we're prepared for big events, like updates or disaster recovery measures or security breaches. But it can be harder to prep for the little hiccups that plunge a routine day into panic. Maybe it's the sheer surprise of the unexpected that makes these moments so memorable.

At the time of this story, I worked as a tech for a large private college. The tech department consisted of a grand total of two employees: the sys admin and myself. My role was as a general troubleshooter for the staff and teachers; I also set up and maintained the virtual servers.

[ For more stories about IT jobs, check out "10 users IT hates to support." | Pick up a $50 American Express Gift Cheque if we publish your tech story: Send it to offtherecord@infoworld.com. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the anonymous Off the Record newsletter. ]

I'd been working there for about a year when the sys admin went on a long service leave, which meant I ran the show by myself until his return. I didn't mind taking on both roles, although at times the extra work piled up.

One day, I went to the college's server room to do some updates and tidy the place, as it was in a bit of a shambles. All went well for a while, and I chipped away at the mess.

I moved on to straightening some network cables, which ran behind a moveable server rack. This rack held several servers, a box containing several virtual servers, and a large SAN. I followed the cables up as far as I could, but couldn't squeeze in between the rack and the wall to get at some connections.

I carefully and slowly moved the rack just enough so that I could fit in behind it. As I wiggled into the now-bigger space and reached for the connections, I suddenly noticed a change in the server room -- an alarming change.

I couldn't put my finger on it and came out from behind the rack to investigate. Then with the stab of dread shared by techs everywhere in this situation, I realized what it was: Silence.

The ringing of a phone from down the hall pierced the eerie quiet of the room and jolted me out of my daze. I went back to the rack and wiggled in behind it, hoping I could restore the tech hum in short order.

I breathed a sigh of relief when I found the problem: When I had moved the rack, the power board at the back unplugged from the outlet. I plugged in the board, tested that all the connecters were set, and restarted all the servers. The server room sprang back to life. Crisis fixed, if not averted.

It took me about 30 minutes to get all systems back up and running and tested. From the tech side, no damage was done and there was no loss of data.

As far as users' work, luckily this happened at the end of the day. Most people either didn't notice anything went wrong or were able to connect quickly enough that they didn't feel the need to report the problem. I had only a few phone calls from admin staff asking about their software not connecting; I told them we had some Internet issues, but they were now fixed.

To prevent such an event from happening again, I made sure the power board that caused all the issues was properly attached to the rack -- where it should have been in the first place. And I left for the day thankful that it hadn't been anything worse.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we'll send you a $50 American Express Gift Cheque.

This story, "When servers go silent: An admin's nightmare," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.