The last working day of the month would find me alone in the office burning the midnight oil. I had to slog through each location's information separately and would be logged into six PCs connected to six different locations at a time. These connections were made by dial-up phone lines with 56k modems.
Always expecting the worst, my first action would be to make a backup copy of each location's end-of-month data in a secure, allocated space on that site's server hard drive. I had started doing this because on rare occasions the closing process would lock up and require a complete restore.
We were in the midst of an internal audit in September and needed to restore the database from a certain remote location from the prior April in order to put together a report. I maintained an old server, not on the network, expressly for such times and thought I should be fine. I brought the server up and mounted the tape.
The tape software required me to first inventory the tape to get its ID, then run a catalog operation on it to list its backups and dates. Looking through the tape catalog I had expected to see a backup from a date for the last working day of April. Imagine my surprise to only see backups from the middle of the month of April. I grabbed the tape for March, and it too had a backup from the middle of that month.
I was stumped. I'd never gotten an error message and was perplexed as to what could have happened. I thought through the process again. Slowly, it dawned on me what had probably been happening.
I surmised that the employee at that location would remove the end-of-month tape and, rather than mailing it, would put it back in rotation before putting that day's tape into the drive. Then they'd grab another tape out of the rotation to send to us, replacing that one with a blank tape. Their system ensured we received a backup, but it was for the wrong part of the month's work. I had no end-of-the month backups from this location.
I sat there for a moment, in shock and panic. I wondered how long this had gone on and if it existed at any other locations. With an IT staff that was already shorthanded and working 50 to 60 hours each week, tape verification was done by reading the log from the tape software, not waiting until the tape arrived at HQ and mounted to check if it was the correct tape. I wondered how I could recover from this one.
Time for Plan B
I switched back to the problem at hand, relieved to remember my end-of-month "safety" procedure to always make a hard drive backup of each location prior to commencing the database compacting and repair. Each location's last-day-of-month tape backups were of the complete hard drive, including my extra backup. I grabbed the May tape, which was also from the middle of the month, and burrowed to the secure area. Sure enough, there was the extra end-of-month April backup. I was saved!
I talked to that employee the next day, and my conjectures were right -- the person said they thought it didn't matter which tape from the rotation they sent to us. You can be sure that we went through the steps once more and that I double-checked the location's tapes for some time afterward.
There was a nice surprise from the situation. This event gave me more ammo to hire another employee and establish additional backup verifications -- finally. I guess sometimes you need a close call of a worst-case scenario to get what you need.
Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to firstname.lastname@example.org. If we publish it, we'll send you a $50 American Express gift cheque.
This story, "How many backups are enough? One more than you planned for," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.