Server upgrade with no upside

The problem was obvious: Data storage was maxed out, the network crawled, and half the time our backups were corrupt. The solution was simple, too: Upgrade the servers and increase network function -- without a loss of data. But sometimes “obvious” and “simple” add up to confusion and catastrophe.

I was head of IT for a small company in Tennessee, and I knew how business worked. So first I made a needs assessment, requested recommendations from experts, gathered proposals, reviewed them, and passed them on to my boss, the owner/CFO and Solitary-Decision-Maker (let’s call him Keith). He told me to start the upgrade immediately while he looked over the fine print.

Fine print, indeed! While Keith stalled, started, stopped, changed his mind, and started over, nearly a year passed. Proposals went past their contract dates and had to be rebid. Vendors were frustrated, and I was stuck patching the outdated, jury-rigged system while I waited for it to self-destruct, taking our data with it.

One morning, when I came in to find smoke drifting out of a server (fortunately it was just a bad fan), I told Keith if he didn’t make a decision, I was out of there. He paled and made the call. Unfortunately, it was the wrong call, at least in my opinion; he put the upgrade in the hands of a company (let’s call it “Dubious Network Consulting”) that I had warned him about. But he was the boss.

Dubious sent Joe -- a “professional” making ten times my salary. Joe was a nice guy, but I got the impression he was a bit out of his depth. I was about to leave for a one-week vacation, so I made him promise that he would not change anything until I got back and he could bring me an external backup that I could check to ensure our data was intact.

I returned from a week in Mexico to discover, to my horror, that Joe had already upgraded several terminals to prepare them for the new server and its new OS. Next thing I knew, a third of our workstations were inoperable, and (surprise!) there was no backup. Our salespeople had no data. Also no calendar, no contacts, no e-mail, no access to documents. All contracts written the previous week were gone. Joe even managed to delete my profile and all my information from the server.

Joe apologized profusely. That was a big help. I called a halt to the fiasco. Then, holding my breath, I began tip-toeing through the data that was left, trying to salvage as much as possible. Fortunately, the boss’s workstation was intact, and I was able to use his hard drive to reclaim many of our current financial figures. Where possible, I restored people’s personal sales information from printed records. Then I signed up the contractor we should have hired in the first place to complete the upgrade.

It was six months before I was confident that we had recovered as well as possible from the data meltdown. But what never recovered was my relationship with our staffers. Every time I had to tell somebody, “I’m sorry, your data is gone,” my reputation sank a little lower. And I didn’t think it would be smart to point my finger at the guy who signed my check.

His money, his call. And my headache. Well, it’s not too late to go into sales.