OK, if you want to hand over one of your most important IT site's infrastructures to a person who just told you he didn't know a router from VCR, fine. Just don't be surprised when something goes wrong.
And go wrong it did. One chilly April morning, I walked into the data center to rotate the backup tapes. In the background over the whine of the air conditioning, I heard a muffled high-pitched "beeeeeeeeeeep" noise. What was it? Maybe the AC?
Before I could investigate, my BlackBerry rang and I was off to take care of an exec's Outlook problem. An hour later, everyone in the building lost all network connectivity. My BlackBerry went nuts with administrative assistants and execs alike calling me desperately to inform me of the outage that I was already aware of.
Literally running, I barged through the data center door and started looking at the network equipment. The real problem was that all of the stuff that was blinking before still appeared to be blinking. Everything looked like it was still working. I called the senior network admin, and begrudgingly he agreed to assist me with the issue, though he was irritated with the fact that I wasn't able to handle this myself.
Over the next hour he guided me through checking connections and doing rudimentary troubleshooting while he juggled other tasks, but to no avail. Frustrated and stressed out, I banged my head on the back of the rack. Then something caught my eye below: One of the UPS units in the bottom of the rack was powered off. Tracing the single power cord connected to it, I discovered that it plugged into a green box labeled "Cisco." Its lights weren't blinking anymore, either.
I powered the UPS back on, the screeching noise started again, and a red warning light for the battery started flashing. Then the unit powered itself off again. The mystery of the screeching noise was solved.
On a hunch, I took the power cord for the green box and plugged it into another UPS. A minute later, like magic, everything started working again and the site was back up. As the networking person on the phone explained, "I guess that would do it. That was the core router for the site."
The director of IT congratulated me on resolving the issue, still self-assured that dumping this responsibility on local IT was the right decision.
A short while later, I accepted a desktop support position at another company. The director of IT eventually made a stupid move that resulted in a major lawsuit for the company, and he was shown the door.
Execs, please take note: Having the right people for the right job is way too important to take for granted. Also, being too aggressive with cost cutting may make the bean counters happy but eventually will blow up in your face.
Do you have a tech story to share? Send it to email@example.com. If we publish it, you'll receive a $50 American Express gift cheque.
This story, "IT sweatshop pays the price for budget cuts," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.