Server smackdown: A networking newbie steps up

IT rookie performs disaster recovery with servers in the hosting facility, then with his career in the CEO's office after a tense all-nighter

This humbling tale comes from a balmy summer evening in 2007 during my first real corporate IT job.

At the time, getting hands-on corporate IT experience was very important to me, so I was willing to work for peanuts. I'd found a position that allowed me to learn on the job as an assistant under the IT manager. Gradually, I would be transitioned into the network admin role as my knowledge grew.

[ For stories about mistakes IT pros have made, read "True tech confessions II: Sinners and winners." | Follow InfoWorld's Off the Record on Twitter for tech's war stories, career takes, and off-the-wall news. | Subscribe to the Off the Record newsletter for your weekly dose of workplace shenanigans. ]

Off the Record submissions

I soon found out that the CEO was tough to work for. Turnover was high, and the IT manager quit about a month after I started. They asked me to assume networking duties while they searched for a replacement. The replacement never materialized, and they asked me to take on the job permanently.

About three months later, one of the production SQL servers at our hosting facility had a bad hard drive that needed to be swapped. The hard drive was one of the two drives that made up the RAID 1 array for the operating system. I phoned the manufacturer and had a technician dispatched to take care of it.

But after the technician left, the RAID 1 array rebuild process failed. I called the manufacturer back, and a low-level tech reading from a script recommended I attempt to remotely reinitialize the drive again. No dice.

The next troubleshooting phase involved this ugly process:

  • Walk up to the server and power it off.
  • Pull the drive.
  • Power the server back on and allow it to see that the drive is not there.
  • Power it back off.
  • Put the drive back in.
  • Power the server back on.
  • Go into the RAID setup menu and reinitialize the RAID 1 array using the two drives.

The technician explained that what should happen is the RAID controller would recognize the existing array on the drive and rebuild it accordingly.

The hosting facility where the server lived was an hour away. I had other projects I needed to do on servers there, so I scheduled a visit to the facility for the following evening. However, unexpected events came up at the office, and I got a later start than I'd planned.

10 p.m.: I arrived at the secured facility and started my other miscellaneous projects. I was able to get them done in a couple of hours. My eyes grew heavy, but none of that! There was still work to do.

1:27 a.m.: I proceeded with the fix to the RAID issue on the SQL server. I followed the process the technician had outlined -- one slow, bleary-eyed step at a time.

Finally, I got to the final step, and within the RAID menu selected the option to reinitialize the RAID array. Then came the critical question that was absolutely necessary to answer correctly: "What sort of RAID array do you wish to create?" (Cue tense background music.)

In slow motion, my hand hit the down arrow key, highlighted "RAID 0," and hit Enter.

"Are you sure? All data on these drives will be lost." This message gave me pause for a moment, but the lady on the phone had said this process would be OK. So I pressed the Y key and hit Enter again. The deed was done. "Array initializing ..."

This looked like it would be a long process, so I went to the break room for a much needed cup of coffee.

1 2 Page 1
Page 1 of 2