Locked datacenter?! Well, let me get that for you

Incident: This was an audacious bit of hardware thievery, submitted to us by yet another anonymous reader. According to the reader, “We got called in to restore an entire office server farm off of tape because the client said his hard disks were … gone.” It seems that management decided the office manager had to have access to every room in the office -- or rather the office manager had complained so loudly and for so long that management finally got tired of listening to her and gave in.

This operation ran a certain amount of business off-hours on the East Coast because it had customers on the West Coast and in Asia. The night before our hapless reader got sucked into this restore, a repair tech walked into the office with a tool bag, wearing a golf shirt that carried the logo of the organization’s usual computer support company. He said he had to upgrade one of the servers and even had a “work order” with one of the boss’ signatures on it. The office manager glanced at the work order and then, in a hurry to get back to her phone, opened up the server room to this guy. She wasn’t worried because off-hour operations revolved mostly around the phone, not the PCs. Ow.

Geek Bond strolls out about 45 minutes later, smiling and assuring the office manager that everything’s fine now. Then he leaves. A few minutes later, someone happens to try to access e-mail only to find that the server is “down.” The office manager angrily calls the computer support company demanding they send the technician back, only to find out they never sent a technician in the first place. Operations are shut down because the support company says it can’t send anyone until the following morning. Enter storm clouds, stage left.

Before that happens, though, a computer-savvy employee who doubles as the in-house desktop support guru decides to check out the server himself the following morning. No wonder it was down. Hard disks, CPUs and, in some cases, the system RAM has been neatly removed from every server -- and apparently placed in the tech thief’s tool bag.

That’s a big problem when the actual tech from the support company finally shows up because (1) there isn’t much he can do without hardware replacements, and (2) the office manager and the boss start blaming the tech for the problem. This, even though the “work order” was an obvious fake with a signature not even close to that of the actual boss -- something the office manager would have seen if she’d looked closely enough. That conversation escalates to a phone call with the tech’s boss, which leads to a sudden dissolution of the support contract. They wound up calling on our reader’s consulting company because it was one of the few in the area that had spare parts and spare servers. 

Fallout: Two days of downtime while the servers were rebuilt or replaced entirely and then restored off of tape. Nothing happened to the office manager, other than a stern talking-to. But the company wrote up a strong policy detailing who was allowed access to the server room and why. Maybe an APB on an Aston Martin would have been a good idea.

Moral: If you’re protecting something of value, you need more than just a lock. You need to manage the keys.

[ Stupid user index | Trick No. 5: Green is great unless it's due to nausea ]