Note to IT director: Less blame, more skills

Specs and requirements are given to the IT director for a critical backup system -- a straightforward request. What can go wrong?

I was employee No. 6 in a tech startup during the dot-com boom. At that point, we did not have an IT staff because everyone in the company could set up and maintain the small network we needed.

But as the company grew to over 100 employees, we built an IT department and hired a director of IT, "Al." Soon after, systems started to get turned over to Al and his team. In many cases, this was an appropriate move.

[ Also on InfoWorld: Batten down the hatches, there's an IT rogue on the loose! Here's how to spot admins gone bad and how to minimize the fallout. | Get a new tech tale in your inbox every week in InfoWorld's Off the Record newsletter or follow Off the Record on Twitter. ]

But in other cases, Al didn't have the training or knowledge to do the job, and many of the systems he didn't know much about had been set up and controlled by employees who were still at the company. He insisted on having control of those as well. Also, it soon became apparent that Al was quick to blame others when problems occurred, rather than helping to solve them. Tensions between Al and other departments started to grow.

Off the Record submissions

My specialty was marketing communications, and we'd set up a website that provided a variety of information to help build our brand name and collect a database of prospective customers. The operation of the website was turned over to IT, but I maintained the title of "Webmaster" to handle direct communications with our site visitors.

One business concern was the database of prospects. In those days, we spent more than a quarter of a million dollars designing and building the website, and another quarter-million or so per month attracting new visitors to the site so that we could increase the database. One critical part of the database spec was the backup. We gave Al's team the technical requirements, they chose the technology for the backup, and we took Al's assurances that the backups were being done.

Late one evening, I received a panicked call from someone else in the marketing department. We were still in startup mode, so people worked very long hours and the office was still full. I drove back to find Al calling me out to everyone he encountered in the office.

"Frank crashed the website! Frank crashed the website!" he repeated over and over. He tried to reboot the servers, but could not get our custom database system working. With all we were spending on banner ads and other marketing to attract traffic to the website, we were wasting money every minute the site was down.

Instead of fixing the problem, Al composed a rambling email to the CEO identifying my database specification as the problem. But that gave me the opportunity to look at the server logs and try to figure out what had happened. It only took a minute to see the problem -- the hard drive designated for the database was completely full, so no more data could be written to it.

Now this was a puzzle. I knew we had millions of records in different sections of the database, but the capacity of the system should have been able to handle tens of millions. A quick look at the directory of the hard drive verified that the master database wasn't the problem.

A closer look revealed the issue; there were dozens of files that weren't supposed to be on the drive but had taken up almost all of the remaining drive space. A quick inspection of those files yielded the answer: These were the backup files!

Al had done exactly as we'd requested; he had written a routine that automatically backed up the entire database every night -- but on the same drive! The backups were numbered so that they were not overwritten, and every day's new data ate up more of the space on that drive.

It only took a few minutes to delete the extraneous backups and get the website on track. It took another day to spec a backup system that even Al couldn't mismanage. But his pattern of blame didn't change -- any problem that existed was caused by someone else, and relations between IT and the rest of the company were strained for months until Al was finally let go.

I still encounter people who wonder why I'm so precise with my technical requirements, down to specifying that backups must be made on an external device in a separate location. If only they knew ...

Get a $50 American Express gift cheque if we publish your tech experiences. Send your story of a lesson learned, of dealing with frustrating coworkers or end-users, or a story that illustrates a relevant takeaway to today's IT profession to offtherecord@infoworld.com.

This story, "Note to IT director: Less blame, more skills," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies