Too many cooks whip up a networking mess

Cheap bosses and bad decisions add up to a networking disaster, and one tech pro has to undo the damage

Too many cooks whip up a networking mess
Reuters/Fabrizio Bensch

How many techs are needed to change a lightbulb? At one company, before the task was passed to me, it had taken four. And in our case it wasn’t changing the lightbulb, but rather basic networking as it related to VLANs in a SMB setting (not a complex network).

First of all, a little bit of history regarding this firm: Senior management tried to be as cheap as possible when it came to spending on IT infrastructure equipment and IT personnel. As a result, for many years they maintained a bare-bones in-house IT staff while hiring external consultants.

Problems remained, such as those that come from different consultants working on different parts of the network or servers at different times. There didn’t seem to be any documentation kept or even asked of the vendors, so we ran into a lot of “unknowns” on the network. Over time, this led to confusion and chaos, such as the notable example of automation equipment coming to a full stop because of an IP address conflict.

Because of this incident, which led to a major loss of revenue, senior management decided to loosen the purse strings and fix the problem with the systems, particularly the network.

You would think that senior management would recruit a seasoned network engineer to look into the problems and address them. Instead, the executive overseeing IT decided to hire a buddy of his to lead this project. The problem? This person was not a networking guy.

During the period when my predecessor was handing over the job to me, he told me about this history and wondered why the executive hired this person to lead, when he felt he could have done the same task. This situation had led to major head-butting between my predecessor and the consultant, who was paid quite a bit more.

It was true that this consultant was not the best person for the job. Because he didn’t know much about what needed to happen, he ended up subcontracting much of the work to a remote Cisco guy in another country. Why? Because he couldn’t afford to subcontract it to a Cisco tech in the United States.

The subcontractor was pretty good with Cisco gear, but he was only able to perform tasks remotely. Unfortunately, there are certain elements of a job you need to see in person to determine the system's architecture -- for example, how the business operates.

Among the IT executive, my predecessor, the consultant, and the networking subcontractor, they addressed some of the issues, but quite a few remained when I got hired. Imagine walking into a kitchen with several head chefs competing with each other. Any guesses on the result? To put it mildly, there was a total lack of coordination.

We had Cisco routers and switches with many VLANs -- too many, in my opinion. Some of the VLANs made no sense given the traffic pattern. Because of the lack of coordination and knowledge-sharing, I faced switches with many VLANs with no labeling or color-coding and, thus, no help for junior IT staff trying to figure out which ports were used for what. It turned into a process of trial and error -- the help desk staffer didn’t know how to log into the Cisco switches and check which ports went into what VLANs.

In addition, the network cabling was a mess -- many of the VLAN ports were not grouped together and seemed to be arranged randomly. I guess that’s one effect of a consultant who doesn’t know networking well, instructing a Cisco tech on configuration.

Due to my previous experience, I was able to fix many of the issues, and the contractor and subcontractor moved on to other items. If I hadn’t been tasked with cleaning up all the remaining issues passed to me, I would have laughed at the number of “techs” required to plug, say, a VoIP phone into the right VLAN on our switches. One day, maybe I will.