I swear I can count!

The embarrassing case of the missing network card

After a decade of being a Sun SPARC administrator, I never thought I would see the day that I forgot how to count.

I had been working on a long, tough project: getting a fairly complex Linux NFS cluster online. The architecture was completely new to me, but hey, it was Unix-based so it had to be good, right? We were using six screaming-fast Dell PC blades (1U's) with high-end CPUs running a lean and clean Linux kernel. The six nodes were to be front-ended by two lower-end PC servers, which would do the NFS handoff to the clients. All I needed were some additional network cards to finish up the base configuration.

[ There are few jobs as, um, interesting as being in IT. InfoWorld has collected some of the most memorable 2008 experiences from the IT trenches | Do you know your geek IQ? Take our quiz and find out ]

After one particularly stressful day, I received my long-awaited dual-port, gigabit network cards. I held them like cherished gold bars and headed up to the datacenter, then carefully removed each of the 1U PC servers and plugged in the cards. I loved this new blade form factor. Remembering my old Ultra 2 days, when removing a network blanking plate too hastily could result in a deep gash or stitches, I thought to myself, "Wow, not bad. No tools needed, and I didn't even draw blood."

Satisfied with my good work and the ease with which it was done, I headed down to my office to begin the configuration. The network team was there to help me set up redundancy to the switches.

Because Linux has the annoying quirk of enumerating network interfaces in an ad hoc manner, I would have to set up the interfaces at the physical level and go back and assign each to a MAC later. So my network teammate and I headed up to the datacenter to sort out which network port went to which switch.

We went to the back of the cabinet, and I counted them off. Server one, ports one and two -- check. Port one connects to server two for heartbeat -- check. Ports three, four, five, and six go to LAN/public -- check. Server two, ports one and two -- check. Port one for crossover and three, four ... uh oh. I looked at the server again. Where are ports five and six? My heart sank.

My network teammate looked at me. I looked at him. I looked at the floor. I knew I'd installed four network cards! I counted again, squinting harshly at my densely packed rack. I figured we were both getting old and must have just missed it. No such luck. My teammate patted me on the shoulder and said, "Let me know when you find it."

I rushed out to the trash bin and found the box the network cards came in: four plastic cases, all empty. I figured maybe someone in the lab found the fourth card and used it. I sent out an all-hands to my trusty lab friends, asking if anyone had seen it. I was willing to suck up my pride and admit to a silly mistake, but this time to no avail. I had to send an e-mail to my boss, who was forgiving and ordered a new card post-haste.

When the new card arrived, my boss handed it to me and said with a grin,"Don't lose this one." I marched up to the datacenter and pulled the server that seemed to have swallowed my card out of the rack. I popped the cover, and looked incredulously into the guts. There was the missing network card, staring back at me.

I can count. However, I had not remembered to remove the blanking plate before installing the card. I'd managed to install the card with the blanking plate in place (both the Intel card's ports and the blanking plate sat flush with the chassis). Damn that sly Intel engineer, made a fool of me he did.

Back downstairs I went to face my boss. Staring at the floor, I handed him the leftover network card. He asked, "What, wrong card?"

I explained and then proceeded to slink back to my gopher home under my pile of shame. I had been doing hardware for a decade. This was the first time I had made such a faux pas.

My lesson learned: When you are not familiar with the hardware you're working with, ask a teammate to work with you the first time -- no matter how menial the task. An extra set of eyes may save you some pride, some pain, and a whole lot of aggravation.

Do you have an IT war story or lesson learned? Submit it to InfoWorld's Off the Record. If we publish your story, we'll send you a $50 American Express gift card.


Copyright © 2009 IDG Communications, Inc.