Stupid QA tricks: Colossal testing oversights

What do you get when you add the human propensity to screw stuff up to the building of large-scale IT systems? What the military calls the force-multiplier effect -- and the need for a cadre of top-notch QA engineers.

After all, if left unchecked, one person's slip of the mouse can quickly turn into weeks of lost work, months of missing e-mails, or, in the worst cases, whole companies going bankrupt. And with IT infused in every aspect of business, doesn't it pay to take quality assurance seriously?

[ For more IT-based idiocy, check out "Stupid user tricks 3: IT admin follies" and "Stupid hacker tricks, part two: The folly of youth" ]

Let's face it. Everybody makes mistakes. Users, managers, admins -- no one is immune to the colossally stupid IT miscue now and again. But when a fat-fingered script or a poor security practice goes unnoticed all the way through development and into production, the unsung heroes of IT, the QA engineers, take a very embarrassing center stage.

It may seem cliché, but your IT development chain is only as strong as its weakest link. You better hope that weakest link isn't your QA team, as these five colossal testing oversights attest.

Code "typo" hides high risk of credit derivative
Testing oversight: Bug in financial risk assessment code
Consequence: Institutional investors are led to believe high-risk credit derivatives are highly desirable AAA-rated investments.

Here's the kind of story we're not hearing much about these days despite our present economic turmoil.

According to a report published last May in the Financial Times, Moody's inadvertently overrated about $4 billion worth of debt instruments known as CPDOs (constant proportion debt obligations), due to a bug in its software. The company, which rates a wide variety of government bonds and obligation debts, underplayed the level of risk to investors as a result of the bug, a glitch that may have contributed to substantial investment losses among today's reeling financial institutions.

CPDOs were sold to large institutional investors beginning in 2006, during the height of the financial bubble, with promises of high returns -- nearly 10 times those of prime European mortgage-backed bonds -- at very little risk.

Internal Moody's documents reviewed by reporters from the Financial Times, however, indicated that senior staff at Moody's were aware in February 2007 that a glitch in some computer models rated CPDOs as much as 3.5 levels higher in the Moody's metric than they should have been. As a result, Moody's advertised CPDOs as significantly less risky than they actually were until the ratings were corrected in early 2008.