"Frank crashed the website! Frank crashed the website!" he repeated over and over. He tried to reboot the servers, but could not get our custom database system working. With all we were spending on banner ads and other marketing to attract traffic to the website, we were wasting money every minute the site was down.
Instead of fixing the problem, Al composed a rambling email to the CEO identifying my database specification as the problem. But that gave me the opportunity to look at the server logs and try to figure out what had happened. It only took a minute to see the problem -- the hard drive designated for the database was completely full, so no more data could be written to it.
Now this was a puzzle. I knew we had millions of records in different sections of the database, but the capacity of the system should have been able to handle tens of millions. A quick look at the directory of the hard drive verified that the master database wasn't the problem.
A closer look revealed the issue; there were dozens of files that weren't supposed to be on the drive but had taken up almost all of the remaining drive space. A quick inspection of those files yielded the answer: These were the backup files!
Al had done exactly as we'd requested; he had written a routine that automatically backed up the entire database every night -- but on the same drive! The backups were numbered so that they were not overwritten, and every day's new data ate up more of the space on that drive.
It only took a few minutes to delete the extraneous backups and get the website on track. It took another day to spec a backup system that even Al couldn't mismanage. But his pattern of blame didn't change -- any problem that existed was caused by someone else, and relations between IT and the rest of the company were strained for months until Al was finally let go.
I still encounter people who wonder why I'm so precise with my technical requirements, down to specifying that backups must be made on an external device in a separate location. If only they knew ...
Get a $50 American Express gift cheque if we publish your tech experiences. Send your story of a lesson learned, of dealing with frustrating coworkers or end-users, or a story that illustrates a relevant takeaway to today's IT profession to email@example.com.
This story, "Note to IT director: Less blame, more skills," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.