Looking for bugs in all the wrong places

Say it ain't so! But even tried-and-true systems running smoothly for years can turn up unexpected bugs when scrutinized

Where there's smoke, there's fire, and where there's software, there must be bugs. Though we like to think the work we do as programmers is important and we're the cornerstones of the industry, we make mistakes like everyone else. It's just a matter of time before they're discovered.

Time was, in fact, of the essence, when I worked for about eight years in a facility with a large data center that was home to a large travel reservation system. Reservation agents worked around the clock and used this system to book travel for car rentals, hotel rooms, and airline seats.

[ Pick up a $50 American Express gift cheque if we publish your story: Send it to offtherecord@infoworld.com. | Get a dose of workplace shenanigans -- follow Off the Record on Twitter. | For a quick, smart take on the news you'll be talking about, subscribe to the InfoWorld TechBrief newsletter. ]

Like many other companies at the time, we put in a lot of effort upgrading systems and running tests to verify that no problems would occur related to Y2K. It was a boom time for contractors and vendors. Many systems were upgraded, not because of actual problems, but simply because software and hardware vendors typically would not certify old products as being Y2K-compliant.

Tonight we're gonna program like it's 1999

The last evening of 1999 arrived, and I was assigned to be at the office for that shift. We had a large contingent of IT people prepared to identify and resolve any problems that might occur. Another shift was prepared to take over in the morning if there was a need. Jan. 1 was a Saturday, so most of us would not normally be working during this time.

The first part of the shift was mostly uneventful. We snacked on food that had been brought in and shot the breeze. Just before midnight, we manned our stations and set about the task of verifying the systems and software for which we had responsibility.

We counted down the last 10 seconds to issue in the new year and held our breath, our eyes glued to the systems. Midnight came and went. Nothing broke, blew up, crashed. The world did not end. The systems changed over just fine for the most part. I am aware of only one problem that occurred that was actually related to the year 2000, and the team that worked with that software spent several hours working to correct it. All our preparation efforts had paid off.

But with so many programmers and IT personnel closely scrutinizing every last detail, it wasn't too surprising that we found other problems that had not been previously reported. For instance, we noticed the reservation system had a problem that prevented the users from displaying airline seat availability for the first hour after midnight. After the hour was up, everything worked normally.

The hour of truth

One problem prevented the software from working properly for the first hour of every year. The other problem had a similar effect for the first hour of every month. These bugs had been in the system for years! We found out later that the reservation agents using this application had noticed the problem, but simply tried again after 1 a.m., when the system worked properly, and never reported it.

1 2 Page 1