A memorable IT meltdown -- for the wrong reasons

An IT manager's 'hostile termination' plus one missed detail create an unbelievable situation

Life and work are full of savory and not-so-savory tales, and IT is no exception. Sometimes a situation simply leaves you shaking your head in disbelief.

Several years ago, my Thanksgiving weekend was ruined by an emergency disaster recovery marathon for a "self-inflicted catastrophe" at a nearby financial institution. My company provided consulting services for Windows Server and Exchange networks, and as the geographically closest consultant in the phone book, we received a frantic call from the CEO on Wednesday afternoon.

[ Do you know your geek IQ? Take our quiz and find out ]

The details of the company culture and employee performance were understandably private, but we were told enough to get us up to speed on why our services were suddenly needed.

The IT manager had been given his walking papers that morning, and while the farewell meeting was underway, a member of the network management team had been busy disabling the user's accounts, resetting passwords on all privileged logins, etc. He'd done a fairly thorough job, but all it takes is one missed detail.

When the (now-disgruntled) former employee arrived at his home after being escorted from the building, he'd found that his VPN connection and remote desktop sessions to all production servers were still up and running on his home PC. Not being one to hold a grudge, he chose to re-reset the Directory Services Restore password on all servers, re-reset all admin account passwords, then began removing user accounts left and right.

When people returned from lunch break, they could not unlock their PCs or log in, and their now-disabled e-mail accounts bounced messages back to all senders. One can only imagine the chaos.

After our team arrived on site and determined the extent of the damage, we exhausted all "easier" solutions and had to resort to using CD-boot parallel installs of Windows on each server. Naturally, there were no records of the exact patch level on each system, so we had to go with the best guess to match the last full backup conditions. At least the latest backup media were not in the drives (and subject to erasure) at the time the rampage was under way, but it still took a very long weekend to get people back in business.

It was a memorable Thanksgiving for the wrong reason -- the actions of one Turkey left a bad taste in my mouth. But since the FBI and FDIC frowned on such goings-on, the perp has spent several Thanksgivings enjoying federally provided meals.

The situation is certainly a reminder that management needs to understand enough about IT operations and vulnerabilities to ensure that the checklist for "hostile termination" of administrators is complete and correct before pulling the trigger. And it never hurts to have a second person double-check.

Do you have an IT tale when something went right, a war story, or a lesson learned? Submit it to InfoWorld's Off the Record. If we publish your story, we'll send you a $50 American Express gift card.


Copyright © 2009 IDG Communications, Inc.