But around 9 a.m., we started getting complaints about Internet and phone problems. We checked cables, network switch ports, and switch and router configurations. Finally, we came to the conclusion that there was another Cisco configuration we'd missed. The switch our Internet connection had been using for more than a year was now sitting unused on a shelf, and we decided to put it back in service. We put it in our rack and made sure to wire all the devices to the same ports they'd been working on before the move.
We breathed a sigh of relief that the Internet and phones were working better than they had been. We decided to leave it that way overnight and call our Cisco expert the next day and get the proper configuration on the switch ports we wanted to use.
Tuesday: Our Cisco expert looked at the switch configurations from his remote office and didn't see a problem. But the Internet, phone, and intranet connection were still spotty, although they were up more than down. By the end of the day it seemed stable, but we weren't sure it would last. And, of course, people were complaining.
Wednesday: The problems continued, so we contacted our ISP at about 9 a.m. to see if they noticed errors on their end. We were down most of the day, with a few up times. A tech finally called back at about 8 p.m. and agreed to replace our fiber-to-copper converter. I left at about 9 p.m., but my coworker waited for the tech to arrive.
Thursday: I arrived to find a note on my desk from my coworker saying he was leaving after 3 a.m. The tech hadn't arrived with the new fiber-to-copper converter until about 2 a.m. The new one was DOA, so a different tech would arrive during the first shift with a working unit.
The new tech arrived at about 9 a.m., and it took him 30 minutes to install and configure the converter. The Internet came up within seconds, and the phones and intranet also worked. Success! Or so we hoped.
Thursday afternoon to Monday morning: No reports of Internet problems. We'd solved it! We figured that the fiber-to-copper converter must have gotten damaged during the move.
Monday afternoon through Tuesday: Intermittent phone and Internet problems yet again. We went through the possibilities again but were out of ideas on how to fix it.
Wednesday: We contacted the Cisco partner we work with, and one of their senior techs came out to look. He wasn't able to find a problem that would explain what we were experiencing.
Thursday: My coworker and I looked through everything for the umpteenth time. What had we not tried? Could it be the new fiber cables? We called the cable installers and asked them to retest the cables.
Friday: A few minutes before 7 a.m., the cable installers arrived, but it wasn't the same team that had installed the cables. It took them about a minute to diagnose the problem: The fiber run to the basement was multi-mode fiber, and we needed single-mode.
I looked again at the original proposal, and it did specify multi-mode. Oops! Was the fault mine or the salesman's who had created the proposal? At this point we didn't care. We'd lost more money in downtime than it would cost to put a new cable in.
We made an agreement with the salesman that we'd pay for labor and the installation company would pay for the materials. And I wanted to replace everything, so we also placed an order for new single-mode patch cables.
Monday (about two weeks after the move): The installers arrived at 6 a.m. The cable was run to the basement and terminated in just more than two hours. By noon, UPS delivered our new patch cables, and we got them installed in the basement and in our space. We then did a switch to the new cable, and within seconds the Internet, intranet, and phones were working!
Several weeks later, we're still up. And no one has complained about the Internet connection, phones, or intranet. Maybe it's fixed this time....
Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to firstname.lastname@example.org. If we publish it, we'll send you a $50 American Express Gift Cheque.
This story, "A missed detail derails a server room move," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.