Dealing with random IT trouble

In day-to-day IT management, much of the focus is on procedure, planning, documentation, and performance metrics — noble things, of course, but they take IT only so far. In my experience, one characteristic distinguishes high-performing IT organizations from merely good ones, and it has less to do with technology planning than it does with the approach to problems. Underneath it all, the most important ingredient for success is a relentless drive to find order within randomness.

And randomness rules. Things tend not to work as expected despite the best efforts of the most able IT department, in part due to the speed at which most organizations must move. Sure, you can (and should) carefully analyze problems, plan solutions, implement carefully, and test as much as is feasible. But the complexity of today’s IT environments means you should always be ready for surprises.

Just this week, we had a classic situation that went a bit beyond the usual IT debugging. Like many businesses, InfoWorld contracted during the downturn and consolidated office space to soldier through tough economic times. As the economy has improved, InfoWorld has begun hiring again, which has led to new challenges. Where do we put these new people? How do we hook them into our network, and most importantly, how do we do it quickly? We faced this issue recently when a new employee was assigned a temporary spot away from the rest of our employees.

Fortunately, one technology that matured during the downturn was 802.11, so we didn’t even think about calling the wiring guys. The new employee was seated out of reach of our wired LAN, but he was close enough for a solid wireless connection. Kevin Railsback, our IT manager, set up an old Orinoco access point, confirmed that our new employee could connect to it, and left it at that. And it worked — most of the time.

Shortly after the wireless access point was up and running, we started getting complaints about intermittent failures in connectivity. Intermittent failures are always the most aggravating for all concerned. Kevin did the usual network due diligence — combing through network logs, looking for clues — but he came up empty. When Kevin walked back to the space, he started wondering about interference in the walls until, suddenly, he had a “Eureka!” moment. There was a break room with a small microwave between the access point and the employee. The signal was solid when he arrived, but when he went to the break room and started the microwave, the wireless signal dropped. When the cooking was done, the connection came back.

We didn’t want to deny the hungry masses on that floor their microwave delicacies, so Kevin installed a newer, more powerful access point that could fight its way through the burrito-warming, Wi-Fi-killing microwaves. No one went hungry and we went back to dealing with other IT issues. Problem solved.

Kevin could have stopped there, but he decided to truly nail the problem by doing the smart thing: He documented it. We usually post this information on our internal Weblog, but fortunately for InfoWorld readers, Kevin documented the problem on his new, public InfoWorld Weblog. Finding order within randomness doesn’t do anyone any good if no one knows how you got there. Good thing there are blogs to close that loop.