When servers crash and burn

I work for a bank in Minneapolis, and several years ago my boss decided it was time to upgrade our loan sales software to a client/server system running on Windows. As the one-man IT department, I was given the task of designing the network, specifying hardware, and installing the new system.

I researched hardware manufacturers, compared customer satisfaction, and settled on a well-respected vendor that could provide everything we needed, especially top-notch support and a great price. I was ready to sign on the dotted line.

First, I had to submit my purchase plans to the boss. Instead of the rubber stamp I was expecting, he told me that an acquaintance had recommended a PC vendor I had already decided against. I disagreed and explained why. He didn’t care; he ordered me to buy from the other guy’s company. It felt like a slap in the face, but I complied.

A few weeks later it was time for off-site training on the new software. But when it came time to buy plane tickets, management decided to send one of the loan VPs instead of me.

“You’re just the IT guy,” my boss told me. “These guys have to use it every day. Don’t worry, they’ll bring you all the manuals.” Yeah, right. Turns out the software vendor relied exclusively on customer training for the product; there was no technical manual! I was starting to get nervous.

Two months after we rolled out the system, the server crashed and burned, managing to corrupt several key files before it fried the hard disk. Although there was a backup server, those corrupt files were copied over before the system went down. Valid users were locked out when their log-in data was lost, and I had never been trained on how to add users to the software. Critical time was lost while I burned up the phone lines with software support.

Next, I had to get the failed hardware back online. Although the nature of the problem was pretty obvious, I still had to negotiate with tech support because it was a warranty issue. Hours passed as the Bengali technician on the other end stepped me through endless script-prompted suggestions that I could barely understand due to her heavy accent. I gritted my teeth, thinking wistfully of the excellent service my chosen vendor was famous for.

Ultimately, she sent the replacement hardware, but by the time it arrived several days later, my boss was breathing down my neck. I finished reinstalling the software at 3 a.m. The next morning, after three hours of sleep, I was called to the front office for a brutal fault-finding exercise. Why had disaster recovery taken so long? When I pointed out that I’d argued against buying the hardware in the first place and that I was supporting software for which no documentation existed, they accused me of making excuses. At least they didn’t fire me.

Finally, a month ago, someone decided that it was time to replace the entire system. The same boss who forced the faulty systems on me in the first place called me into his office and said, “Do a little more research this time. Those machines you installed weren’t very reliable.” I thought about my mortgage and decided not to murder him.

When nontechnical managers make technical decisions, IT people always pay the price.