I finally got a call from a support person -- not the one assigned to work with me, but someone the company found. We got started and discovered that the switches were not managed. The support person informed me there were old hubs in the warehouse that had been removed during an upgrade a few years prior. He wanted me to put one of them back into service and get a few workstations up and running until new switches could be shipped to the site. He said he would ship out new switches to arrive the next day.
I found the hubs after looking around the warehouse a bit and was able to set them on the rack in a way that sufficed for temporary use. However, I ran into a problem: None of the jacks in the building were labeled to the patch panel, and they had more than 100 drops. I managed to tone out 4 or 5 of the important ones and powered on the malfunctioning switches to see which jacks had equipment plugged into them. In the process, the problem switches started working just fine.
I called the new support contact to tell him the switches were working after all. He said they still wanted to change from nonmanaged to managed switches, and he asked me to come back the next day to do the upgrade. I also asked for approval to clean up the wiring, and he said he'd send the request up the food chain.
I returned the next day, opened the package with the new switches, and got started. I hadn't received approval to clean up the wiring, so I was focused on the switches. But the disarray was so great that I ended up having to clean enough of the wiring just to be able to pull out the old switches. In the process, I discovered a total of three patch cables plugged into a problem switch that touched the ground and went back into another port on the same switch, causing a broadcast storm that had brought down the whole network.
The three days of downtime at this site must have cost the company tens of thousands of dollars in lost productivity and extra help desk fees, all because of a messy rack and far-flung tech support.
I've worked at many sites and different companies, and it seems that the organizations with better IT budgets and better-organized IT departments normally run better overall. This company may have saved time and money in the short term by leaving its cables in disarray, but the messiness took a large monetary toll when repairs were needed. The details matter, even down to the orderliness of a network rack.
Do you have a tech story to share? Send it to firstname.lastname@example.org, and if we publish it, you'll receive a $50 American Express gift cheque.
This story, "Sloppiness in the server room brings down the network," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.