Who's doing the backups? Even IT guys get it wrong

An IT tale of how something as important and basic as backups can slip through the cracks

A question often asked is, "how could anyone be so stupid as to not back up data?" Reflecting on my own experience, I see how it could happen. Sometimes a series of events can interfere with important and seemingly basic IT tasks, such as backups.

I work as a software engineer for a small company that was formed over 10 years ago when our founding CEO, "Ed," got fed up working for an employer who thought that paying the employees was optional. One of our biggest problems from the beginning has been backups.

[ Want to cash in on your IT experiences? InfoWorld is looking for an amazing or amusing IT adventure, lesson learned, war story from the trenches, or an instance when something went very right. Send your story to offtherecord@infoworld.com. If we publish it, we'll send you a $50 American Express gift cheque. ]

Our company started in Ed's basement with a few people who were tired of working for promises that never materialized. Ed got a contract writing software as a consultant and split his paycheck with the other employees while they tried to find other work. Our business grew. In what little spare time we had, the engineers would work on internal IT needs. But for the most part we were a bunch of tech-savvy software guys who were too busy "getting the job done" to worry about backups right then.

Ed thought that he knew more about IT than any IT person he'd hire (and to be fair, he did know quite a bit). Eventually, though, Ed reluctantly agreed that we needed and could afford a full-time employee for IT. Being an office full of engineers, many of whom had been around working with computers and writing software since punch cards (and before), our IT needs were probably a little different than most. We were mostly Unix guys and the company couldn't afford anything but Windows PCs, so Ed hired First IT Guy, a young high school graduate with a Microsoft certification.

Backups became an explosive situation between Ed and First IT Guy. Ed recognized that we could be out of business in a heartbeat if disaster struck and that we needed backups. However, he didn't listen to First IT Guy's input on matters, especially if it meant spending more money -- Ed thought that any good IT professional could figure out how to make anything work. Also, First IT Guy had an enormous ego and made it clear that dealing with such menial things like backups was beneath him.

Finally, Ed stopped by a computer store on his lunch break and picked up the cheapest backup system on the shelf. First IT Guy messed around with it for a while but never really got it working, partly because we theoretically had a file server and network but were really storing all data on our own computers and only used the network to talk to the printer.

When Ed figured out that we didn't have backups, he confronted First IT Guy about it and was told why the backup system wouldn't work. The next time Ed was at the computer store, he bought another one that was a little more expensive. First IT Guy worked on it for a long time, but still no go.

After about the sixth repetition of the backup fiasco, First IT Guy found another job. One of our engineers got saddled with the IT duties, and all of a sudden our network started staying up for months at a time, our server started storing our data, and we had a mirror of the data on another machine. It wasn't very secure, but an improvement. Ed refused to buy another backup system and insisted that the engineer make one of them work. The engineer did get something to work, sort of, once -- then was too busy with his other job to deal with it.

To keep up with changing needs, we converted most of our boxes to Linux and bought expensive mainstream software. We bought a couple of new servers: one for a mirror, and one with a tape backup system that actually worked and had a support contract from the manufacturer. We finally had backups of all our engineering data. The data for the CEO, accounting, and sales wasn't backed up, but the engineers considered that someone else's problem because we sure didn't have time for it.

Eventually, we hired another young high school graduate who had just finished a Microsoft certification to fill our IT position. We were deficient on backups for the Windows side of the business, and Second IT Guy tried to get it going. Unfortunately, Ed considered his information and the CFO's accounting information too sensitive to be trusted to a lowly IT person. Ed got highly annoyed when anyone mentioned that if he wouldn't let Second IT Guy back it up, he would have to back it up himself. The CFO started backing up her own data, but Ed didn't get around to it.

The situation came to a head when Ed's computer died. He went out and bought parts to fix it himself, but when he couldn't get them to work right he finally consented to let Second IT Guy fix it while he went to lunch. In the process, Second IT Guy accidentally wiped Ed's data.

It wasn't long before we were back to an engineer trying to take care of IT in his spare time. Ed swore that he would never hire another IT admin or let anyone else touch his computer. In time he ended up getting a laptop (or two) and did his own sneakernet between them.

Our engineering server had a hardware failure a short time later. After about 20 minutes, we were back up and running on the mirror, and the next day the manufacturer had a tech on site fixing our server, which was back up the day after.

Unfortunately, a month later we had a major problem. Our very expensive engineering software had a bad habit of creating lock files and leaving them laying around when they should have been deleted. We got used to typing rm -rf *.lck to remove them by hand. One day, one of the engineers typed rm -rf * without the .lck. Our project that we had been working on for months was gone. We then found out that after the main server had died and we'd switched to the mirror, we hadn't switched back. We were able to recover what we had from a month before, but had to redo a month's worth of work.

Since then, we have been able to get updated engineering software and hire someone with a job description of CAD and IT (who does all the IT stuff very well, but since he also does CAD work Ed didn't break his word). We now have mirrors, tape backups, and even off-storage of tape backups. We have a different CEO who encourages CAD/IT Guy to research solutions and isn't afraid to spend enough money to get what is needed.

If disaster strikes, we are -- finally -- ready.

This story, "Who's doing the backups? Even IT guys get it wrong," was originally published at InfoWorld.com.

Copyright © 2009 IDG Communications, Inc.