We asked the customer to diagnose the network problem, and they told us that network administration was the responsibility of another company. We promptly got in touch with them, and they assigned an engineer to troubleshoot the problem.
The engineer told us to run some tests. We had already done these tests, but did them again and got the same results as before. He promised to get back to us.
Days went by. We contacted the engineer again and received no reply. Our company's liaison with the client called the network service provider again. They assigned another engineer to diagnose the problem. He requested we perform the same tests as before. We did, and sent him the results -- and got no reply. This same process was repeated with yet another network engineer.
Meanwhile, all testing on the environment had been halted, and the deadline for going into production was postponed. The customer's middle managers were furious and frantic -- postponing meant an incredible headache to them, and they started begging their superiors to mobilize resources to solve the problem. But apparently their superiors didn't care much, and the problem persisted.
Finally, after about a month, the network service provider company gave us their answer: They had no idea what the problem was. All network switch links were operational, and testing showed no signs of a problem. It had to be a problem with the machines, but we'd been told by the customer that the machines were fine. We were baffled.
The customer's top managers still didn't do anything to help figure out what was going on. Our customer liaison was going way beyond the call of duty to get the customer to find out what the problem was with their own machines. By opening unofficial lines of communication with employees working for the customer and by gradually gaining the trust of people in several departments, she was finally able to solve the mystery.
It was simple: The problem machines had been moved from one physical location to another, without our knowledge. They should have been up and operational after two days' delay, but due to the customer's internal bureaucracy, they had been forgotten in a warehouse and were still sitting in their crates waiting to be turned on and plugged into the network. Since the customer's Unix team didn't have problem tickets opened for the machines, they assumed they were in working order, and this they told us without bothering to check.
By the time this was discovered, the customer's testing phase had gone completely off schedule, and services that their company was scheduled to provide to their own customers had to be postponed, risking cancellation or renegotiation of contracts -- all because of a combination of bureaucratic inefficiency and neglect.
From that point on, we became very wary and demanded from our customers proof at every stage of a job. And we learned not to trust anybody ever again, except the people who we knew were also suffering from the same problem. In short, we learned that many times in such large environments only the people directly affected by a problem are genuinely interested in solving it.
Do you have a tech story to share? Send it to firstname.lastname@example.org, and if we publish it, you'll receive a $50 American Express gift cheque.
This story, "The strange case of the missing servers," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.