On Monday, the button arrived, and we made the two-hour drive to the aerospace customer's site.
The system was waiting for us. We unpacked it, and the parts buyer installed the button first thing. Foolishly thinking it was all downhill from there, we plugged in the system, powered it up -- and nothing. It was dead on arrival.
Panicked, we tried to unearth the problem, but we were stumped. Never mind that it had checked out OK before we shipped it -- it definitely wasn't working now. There wasn't even a video carrier coming out of the system, although the power supply was producing the right voltages, the boards were seated, and the fragile wire-wraps were in place.
Finally admitting defeat, we called our company's chief engineer, who did the final stages of manufacturing on all the systems (though in those days nothing was documented). The phone call stretched into hours. Finally, he decided he needed to come to the site himself and left in time to make the trip before the end of the day.
When he arrived, the first thing he did was hook up his logic analyzer. A couple of minutes later he said, "This is strange, it's constantly resetting."
Then he removed the newly installed Reset button. The system came up with no problem. We all were amazed. And relieved. And angry that we'd missed something so simple.
To make a long story short, the supplier of buttons had sent the wrong part number -- the button fit in the same square hole but was "press to disconnect" instead of "press to connect." Unless the button was held down, the system would constantly reset.
I learned a valuable lesson that day, which I express as this homily: Test the way you will deploy. I sometimes add, "Unless failure is an option."
To tell you the truth, faced with the same problem today, I'd probably make the same choice, but with a much better awareness of the risk involved, balancing certain loss of revenue against possible reduction of reputation. But I certainly wouldn't say, "What could go wrong?"
Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to firstname.lastname@example.org. If we publish it, we'll send you a $50 American Express gift cheque.
This story, "Button it! A tiny detail snags a system setup," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.