Further investigation showed that the hardware address of the router appeared to change for a few minutes at a time, then revert to the expected address. To my surprise, opening a terminal session to the router when the hardware address was "wrong" resulted in a log-on prompt from a well-known brand of print server.
This organization had many older printers that had been network-enabled by connecting them to print servers, and now I had to find the one causing problems. The CIO was hovering outside the door and looked puzzled when I explained to him what I was looking for.
We located the print server and discovered that an incorrect network configuration was loaded. We removed the print server from the network, then reconfigured and reconnected it. We checked to see if the connectivity problems were resolved: They were.
We pieced together what had happened. There had been a power brownout in the area earlier that day, which prompted the offending print server to restart. When it did, it loaded a stored network configuration that gave it the same network address as the corporate router. Then it was a matter of luck as to whether a PC tried to use the router or the print server to access servers on the other subnet.
The print server had operated without a hitch for more than three years running a correct configuration, but also storing a second, incorrect configuration. Its luck had run out.
The systems controlling the national grid were never at risk, which was a relief to the CIO, and productivity at the office was restored.
The CIO was appreciative and thanked me for figuring out the problem. But what else could he say when the evidence of the problem ultimately came down to a human error by someone on his staff? I must admit, the end result of this situation was definitely satisfying to my company's team.
This story, "Stand back and let IT do its job," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.