Server smackdown: A networking newbie steps up

IT rookie performs disaster recovery with servers in the hosting facility, then with his career in the CEO's office after a tense all-nighter

Page 2 of 2

2:10 a.m.: I returned to check on the progress of the RAID array and arrived just in time to see the progress bar tick to 100 percent. Cool. I casually powered the server off and back on again. Then it happened.

My blood went cold as the words "Operating system not found" appeared at the top of the screen. I stared for a minute, unsure what to do.

Then it occurred to me: I should have selected RAID 1, Not RAID 0! Curse words could be heard over the whine of the air conditioners.

2:14 a.m.: Remembering that we contracted with a hosting company to do full data backups of our servers, I put in a desperate call for a rush restore.

"Yeah, you see, you have a contract for operating system backups, but you never contracted for 'bare metal' recoveries. Unless you have a functional server, there is nothing we can do for you."

More cursing on my part.

2:21 a.m.: I dumped the contents of my backpack out on the table and found the Windows 2000 Server SP4 disk. Shoving it in the drive, I rebooted. After the obligatory setup questions, the operating system started loading.

Where was my SQL Server 2000 disk, though? More cursing as I realized it was in the other folder back at the office!

2:53 a.m.: I jumped in my car, hit the gas, and made the 1-hour trip in about 35 minutes. Within minutes I'd retrieved the entire CD folder and threw it in the car. Warp 9 back to the hosting facility.

4:07 a.m.: When I got back, the Windows 2000 install process had gone OK and the server had a basic unconfigured copy of the operating system on it. Next, all of the IIS components, updates, and so on had to be installed. Finally, I was ready to install SQL Server 2000.

Fortunately, the databases lived on the RAID 5 drive, which was not blown away. All I had to do was reattach them to the server. I did a test connect from another PC, and the server was OK.

6:01 a.m.: I exited the hosting facility building just in time to see the sun rising over the smoggy horizon. It had been a bad night, but the drama was not over yet. During my slow rush-hour drive back to the office, I had time to contemplate my coming professional demise.

7:27 a.m.: I moped into the office front lobby, a dead man walking. I had killed a production server, it was totally my fault, and I probably deserved to be fired for it.

We were yet again low on staff, so at the time I reported to the CEO. Walking into his office, I shut the door and explained everything that had happened. However, he was only half-listening and probably didn't understand most of what I was telling him anyway. He cut me off midsentence and said, "But it's working now though, right? That's all I care about. Now go home and get some sleep."

I will not lie -- I walked out of the CEO's office smiling, choking back tears at my good fortune.

The server still had a RAID 0 array with no redundancy, but at that point I was afraid to touch the server again to change it back. It wasn't long afterward that the client whose data was stored on the server went to another company for its data hosting. Apparently, the dysfunction of the company was obvious enough in other ways that the client no longer wanted to do business with us. The aging server was backed up one last time and decommissioned.

Within a couple of months, I had found a desktop support position with another organization and moved on myself, a little wiser than when I'd arrived.

Do you have a tech story to share? Send it to offtherecord@infoworld.com. If we publish it, you'll receive a $50 American Express gift cheque.

This story, "Server smackdown: A networking newbie steps up," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

| 1 2 Page 2
From CIO: 8 Free Online Courses to Grow Your Tech Skills
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies