Autopilot may work when you're flying a plane, but not so much when you're minding the network. When your mind is elsewhere, it’s so easy to make a simple mistake with big repercussions . Here’s one such slip I made that involved a high-end client and -- thankfully -- an understanding manager.
It was my first IT job. I was a support technician/trainer for a software company that developed a DOS-based scheduling software for high-end resorts. Needless to say, the software was complicated and used several dozen dBase databases. I was getting pretty good at troubleshooting and fixing the various issues that cropped up, and I grew less dependent on my manager every day.
I had been with the company for about a month when the higher-ups decided to offer extended support hours to accommodate our clients on the West Coast. It was my turn for the late shift, which I was dreading as I hadn’t gotten much sleep due to spending the day with friends.
Simple problem, stupid error
An hour before my shift ended, I received a call from a client (a five-star hotel that was both well-known and very expensive) stating that appointments were disappearing or showing up garbled. I smiled to myself because this was a simple problem that would take about 20 minutes to remedy -- I had fixed similar issues numerous times.
I was finishing up and getting ready to call the client to let them know everything should be good to go and they could start using the application again. I had one more task on my check list: Delete the temporary database files I had used for cleanup.
Feeling elated, tired, and ready for bed, I did the unthinkable: I typed in
del *.* and hit Enter. For those of you who remember old DOS commands, you’re probably cringing as much as I did when I realized what was happening. For others, back in the day of DOS,
del *.* wiped out all files in the directory without prompting for confirmation.
By the time I realized what I had done and canceled the request, more than 15 databases were gone. I was white with shock. I got up from my desk, approached my manager, and explained what I had done. He went whiter than me, his eyes wide as saucers. This made me feel even better -- not.
He walked over to my desk with me and tried a few moves, but the files were gone. The next step awaited, although neither of us wanted to do it. My manager picked up the phone and called the client. He explained that the files were corrupted beyond repair and had to be restored from the latest Novell backup.
Grace under pressure
I stared at my boss a few seconds, then it clicked: Novell didn’t delete files completely at first -- instead, it merely marked them as deleted. I stopped him immediately, telling him we were going to make one more try. He relayed this to the client and put them on hold. I proceeded to explain to my boss my new plan. He agreed, told the client we’d call back, and gave me control of the system.
I located the files in Novell’s deleted.sav directory and restored them -- as both of us breathed a big sigh of relief. We tested the software and everything worked. We called the client to let them know everything was ready for them to get back into the software.
Crisis averted! From then on, our new procedure (and my new mantra) involved making a backup of everything before we even started working. It was a hard lesson to learn, but I never forgot it.