The only thing was, when I reset the system to the current date, it still failed. I started trying to diagnose the problem. The Facilities Manager pitched in, taking the cases off all the computers and vacuuming the dust.
"You'd be surprised what a little dust can do to a computer," he told me.
After about 3 hours on the phone with a programmer who developed some of the components, I determined that they had applied a patch the prior month. I looked in the event logs and discovered that the systems had not been rebooted after the date of the executables. In other words, the developer came in, applied a software update, and did not test it. Did not even reboot the machine. The DC personnel never rebooted the machines, so this one had been running the old code for a month, with the newly installed, untested code waiting for its chance to wreak havoc. In effect, I was doing a test of the code that was installed a month ago, not a Y2K test (oh well). I ended up rebuilding the code from the source archives and installing it on the failing machine.
All this kept me at the DC past 6, so I called Jack, who was already at the bar, and told him I'd be late. He was OK with that.
And then at 7 p.m., the production cycle began. I discovered that the recovery process had reset all the calibration settings on the sorter, causing containers to miss their exit lanes and crash into the sides. Normally, fixing this problem is a two-person job, with one person at the sorter (usually on a ladder) speaking to another in the computer room via walkie-talkie relaying how much the containers overshot or undershot their exit lanes. But that evening I was on my own, running from room to room and climbing up and down the ladder. After what felt like the equivalent of a 5K run and a stair-master session, I finally had things back to normal at 10 and promised to return at 8 the next morning to do a post-mortem in front of the muckity-mucks. I high-tailed it to the bar and found Jack. He had made many friends.
We ended up spending the major portion of the night out on Bourbon St., shooting pool in a little bar on a side street. We played the locals for pitchers of beer, made the obligatory stop at the Café Du Monde for French doughnuts and coffee, and staggered back to the hotel at about 4:30 in the morning. That wakeup call at 6:30 came way too early. I dragged myself back to the DC, somehow managing to make it through the day until the DC manager was satisfied that I had restored his system back to normalcy.
I don't know if I learned anything from this experience other than that at times it's best to distrust other programmers' methodology -- and how nice it is to have good friends.
Do you have an IT war story or lesson learned? Submit it to InfoWorld's Off the Record. If we publish your story, we'll send you a $50 American Express gift card.