Unprepared for hidden costs

A NOSTRUM of the late '90s was to dump internally developed, custom applications in favor of packaged applications. But the hidden costs of those packaged apps are coming back to haunt many companies that implemented them too hastily.

Free IT resource Hear how top CIOs turn change into a competitive advantage. Sponsored by HP Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld Hidden costs of packaged applications Executive Summary: Packaged applications offered many companies a purported solution to the costs of internal development, but complexity and lack of planning for packaged software's hidden costs can end up making them more expensive, costing more time and money than anticipated. Test Center Perspective: Addressing packaged apps' hidden costs takes planning. The five-layer OUCH model can help illuminate costs, ranging from obvious (hardware) to tangential (training time, integration) and organizational (opportunity costs).

The motivation behind that lurching, tectonic shift was the horror generated by over-budget attempts to build systems internally. An entire generation of in-house programmers was pulled from projects and thrown at ERP and CRM systems -- or out the door -- because executives believed that packaged apps would fix the problem of hidden costs.

"Companies that had been building their own software for 40 years started thinking maybe they didn't need all those people, and that was a sign of something bigger: a crisis of confidence," says Tom DeMarco, a Camden, Maine-based principal at Atlantic Systems Guild and co-author of Peopleware.

DeMarco thinks the rush to packaged software was accelerated by organizations losing confidence in their ability to invest wisely in IT and the creeping belief that original works of code were merely a cost, not an investment. But the hidden-costs problem didn't go away with the adoption of packaged apps, which can hide even more expenses.

Tackling the hidden costs of packaged software can also drag down projects and hopes of simple adoption strategies.

"If companies were feeling defeated by custom software, then packaged software has proven an even greater defeat. The fiasco with Oracle last year indicates that even upgrading a package isn't affordable," DeMarco adds, referring to the snarled application-migration problems last August that purportedly cost Agilent more than $100 million.

Uncovering layers of hidden costs

Why do hidden costs seep into every significant software project, custom or packaged? To understand, you have to apply what I like to call the OUCH (Overwhelming Unforeseen Cost Headaches) five-layer model of potential hidden costs to your packaged software deployment project.

Layer I is the set of costs that appears on any vendor's or SI's checklist, including staff, the packaged application, and dedicated hardware.

Layer II, the "absolutely should have known" layer, contains costs that any competent project manager would have accounted for in scheduling -- items such as package customization and post-deployment, dedicated tech support.

Layer III includes the costs tangential to the project's mission and specifics: efforts the organization wouldn't have required without the project, but ones that lie outside the way normal organizations scope deployment projects. Tangential costs include post-deployment training, integration, and the repurposing, firing, and hiring of personnel associated with the application.

Layer IV includes costs that the deployment creates external to the project itself but become an expense to the departments touched by the software package. External costs include changes required to other systems (electronic and manual) and loss of effectiveness during the early part of the learning curve.

Layer V is the organization structure and superstructure layer. It includes project-generated distortions of effective behaviors throughout the organization for political or emotional reasons, and opportunity costs of putting one project ahead of another.

The higher up the stack you go, the more difficult it is to account for the hidden costs. Although internally developed custom software shares the same potential for these costs, most of these unforeseen costs are more controllable in a custom software deployment. Internal critics also have more power to affect an internal project's design than that of a packaged one. The luster, self-confidence, and branding of packaged application vendors help to mute corporate Cassandras.

The fact that these large-scale systems work most coherently when accepted in their entirety, without customization, creates an all-or-nothing proposition that's polarizing, increasing both politics and human engineering stress.

Layers I and II: the obvious

There are obvious costs that should never have been hidden from an experienced observer: upgrades to hardware and operating systems to accommodate new, "more robust" (translation: blubbery) program code, and consultants to tweak and modify the packaged code to match the enterprise's requirements, for example.

"It seems like it's always a surprise to the client how much it costs to customize the package when they find out it doesn't have something they want," says Doug Zeffer, vice president of production at Enlighten, an Ann Arbor, Mich.-based Internet professional services company.

"Buyers take for granted a lot of things, don't ask the right questions, and discover after they bought it the features they requested aren't included," says George Brown, president of Database Solutions in King of Prussia, Pa. "And their sense of the complexity of these solutions, because they're packaged, is too low."

Another issue that can torpedo estimators' plans involves underestimating the amount of effort that it takes even great systems programmers to tweak, customize, and continually upgrade a packaged application. A high-performance coder such as Nik Malenovic, managing director at ThoughtSpring Consulting in Chicago, may spend 10 minutes trying to do a task in a new system it would have taken him 30 seconds to do in a system he already knows.

For Malenovic, the coding skills are so ingrained in his psyche that he has to "translate" when presented with another format, and in that case, "familiarity becomes a liability," he says. That liability sneaks into projects when managers, knowing high-performance coders' usual productivity rates, assume the same high rates on new packaged app tasks. Phillip Gordon, a consultant at Neon Consulting Group in San Francisco, points out an expensive hidden cost in Layer II that involves people rather than technology: critical staff being off-line during training.

"Yes, the vendor told you," Gordon says, "and you even paid for the training and set aside the time in the project plan, but when that time comes, their managers scream because they haven't planned for having their staff out of pocket for three days, not doing scheduled work."

Layers III and IV: revenge of the subtle

An even bigger people cost skulks in Layer III: ongoing training and technical support. "The cost of training -- and it's a significant one -- is an ongoing responsibility, and the cost isn't being tracked," according to AMR Research Vice President Bill Swanton. "There's the training you need long after deployment for new people or those who change positions."

And, of course, there's the lost productivity of users as they abandon their usual way of doing things and climb the learning curve of the packaged app and its new processes.

These hidden costs are much more subtle because of a fatal weakness of the finance professions: Most costs don't fit into a chart of accounts. No CFO can trace the inevitable external cost that Database Solutions' Brown cites: "How do you get focused mind-share from the users in flattened, lean, and mean organizations? They've been re-engineered, they're doing three jobs, and you want them to be effective [while] learning a massive new way of doing their work. They're out of altitude, air speed, and ideas" and will end up underperforming any estimate.

Layer V: the death spiral

Every corporate neurosis festering inside an organization can derail deployments by lurking in the background and loading hidden expenses. And in a co-opetition-filled, merger-frenzied, online-partnershipped world, it doesn't end with internal politics; the inter-organization costs can be astronomical, too.

Enlighten's Zeffer has seen the effects of these hidden costs firsthand. His organization built a CRM system for a billion-dollar homebuilder. They front-loaded the analysis and design, spent a year mapping how the parts of the company would integrate, and crafted it around the company and its culture.

The unforeseen factor was a merger. The client company was the acquirer, but when the two entities merged, the smaller company had a bigger IT group, so the acquired company forced its technical infrastructure on the acquirer. The acquired IT group insisted -- without knowing the analysis and design -- on micromanaging, asking for explanations of every fragment of the system that was already explained and signed-off, delaying its implementation, Zeffer says.

The single biggest hidden cost associated with packaged apps hasn't hit the industry yet, but Atlantic System Guild's DeMarco believes it's inevitable, and it's a killer. He calls it "latent turnover," and it has deep roots.

By 1997, many companies already had a three-year backlog of development work; a backlog that was exacerbated by the pressing needs of immediate, must-do-now initiatives such as Y2K and the Euro conversion. "There's got to be seven years of backlog now," DeMarco says, and the developers who are left must work on the relatively banal tasks of maintaining and buffing packaged vendors' code -- and it is demoralizing.

"The result is latent employee turnover ... they've decided to leave, they're checked out, but they're still there," DeMarco says. "The companies retreating from organizationally discriminating development have left developers disenchanted, but they are not leaving because in this economy there aren't options."

Without a push for changes in packaged applications and their treatment, when the economy does turn around, resources for attacking the gargantuan backlog will become available -- but disenchanted developers, now with other options, will leave in droves. DeMarco sees this putting CTOs in a panic: a sudden upsurge of work for developers but decimated coder ranks.

Also alarming is a different latent turnover that could create another lurching shift. Dave Perkins, principal at American Management Systems in Fairfax, Va., says that because IT has commanded so much of the strategic mind-share for so long, it's not just applications that are backlogged ... strategies are, too. Perkins thinks the passionate focus on packaged enterprise applications begat an "idea backlog," which created the same disgruntlement among top managers as packaged apps have among developers.

Perkins believes the result will be the same: When economic situations make moving on possible, the key strategists will be fleeing their current employers, immobilizing ambitious companies looking to take advantage of a recovery. And that's a cost that will not be easily hidden.