"I got to say, I was pretty popular," he jokes. "I think the only reason they didn't fire me was by that point they had gotten so used to me screwing up, they realized I couldn't do anything right."
Lessons learned? 1. Test the restores, not the backups, says Harry. "No one cares if the backup works; they care if the restore does." 2. Think before you type. 3. Remove your pager (or BlackBerry) before entering the telecom closet, just to be safe.
True IT confession No. 2: Sometimes it takes a janitor to clean up an IT mess
Late one night in 1997, Josh Stephens was working all alone at his console at a large Midwestern telecom company. Stephens was making changes to the Cisco Catalyst switches at the telco's main customer call center, which was located several states away. That's when the spanning tree protocols hit the fan.
"I'm still not sure exactly how I did it, but I caused some sort of broadcast storm and STP freak-out that locked up not only the switch I was working on but every single switch in that facility," he says. That broadcast storm brought down hundreds of call center users, stranding many of them in the middle of customer calls.
[ Of course, janitorial services and IT don't always mix: Server room. Windex. Zot. ]
Worse, the switches were "locked hard," requiring a physical power-off and a slow methodical plan to bring them back online, one at a time. The datacenter was hundreds of miles away and had no on-site IT staff, so Stephens did the next best thing: He called maintenance.
"I ended up finding a janitor that had keys to all of my LAN closets and I talked him through (a) which devices were the Catalyst switches, and (b) how to power them off," he says. "I also promised him he wouldn't get fired for helping me."
Though the call center was down for more than hour, nobody ever found out why or who was behind the glitch, says Stephens, who is now VP of technology and Head Geek (yes, that's the actual title) for SolarWinds, a maker of network management software.
Lessons learned? 1. Don't make changes without scheduling a window for them, even if the changes seem minor, says Stephens. 2. Never conduct a change control event without IT resources near the gear you're changing. 3. Be nice to the janitors. One day they might save your assets.
True IT confession No. 3: Put your hands up and step away from the terminal
One of the unavoidable facts of tech life is that when managers are given administrative rights to complex systems, bad things tend to happen.