I work full-time as an IT manager, and once in a while, I accept little contracts for SMBs when the job is short and the money is good.
About a year ago, I was contacted by "Greg," the computer guy in a construction company, for help migrating a server to VMware. Greg had been given my name by the company's accountant, who had once been a summer student with a company I used to work for. (I found out later that the new server project was the accountant's initiative, and he was not comfortable with Greg's skills.) Since Greg was alone doing IT at the construction company, it was an administrator position even if his knowledge and experience was that of a first-level technician.
[ Want to cash in on your IT experiences? Send your story to firstname.lastname@example.org. If we publish it, we'll send you a $50 American Express gift card. | Show off your geek quotient: Get your InfoWorld shirts, cups, and more schwag. ]
Greg had a powerful server that was used for accounting and wanted to convert its existing OS to a virtual machine and run new virtual servers on it. Because he didn't know Linux or VMware, he asked me to come over and reinstall his server for him. So around 9 a.m. on a Saturday morning, we got started and outlined our plan of attack. I advised him to do a physical-to-virtual conversion, but he preferred to reinstall the server from scratch with the accounting application and restore the database later.
Before starting I asked him to perform backups of his data to make sure we had everything. He said that wasn't necessary, that the backup was likely to have run last night, and he showed me his tapes. I insisted that likely wasn't good enough in IT and that we should do a backup right away to be safe. I guess he was worried about my hourly rate because he refused.
I reluctantly formatted the server and performed a basic Debian install, then installed VMware server. Greg then installed his virtual server and accounting application. When the time came to restore the data from the tapes, it turned out they were empty -- they were like new, untouched.
Unfortunately, Greg had labeled all his tapes from Monday to Friday and set up his backup software to write only the tape for the current day. Everyday, he would put the tape in labeled for that day of the week: Monday for Monday, Tuesday for Tuesday, and so on. The problem was, his backup was running at midnight, which was the next day. For almost a year, none of his backups were successful, and he didn't check his logs. Good admin.
As it turned out, his boss and the accounting team had worked extra hours the week before, cleaning all the financial data and closing the fiscal year. All the previous week's hard work was lost -- and so was the rest of the year's.