Lesson from the meltdown: Listen to your architects

An enterprise architect gives an eyewitness account of IT dysfunction inside Citicorp -- and how that prevented the financial giant from perceiving huge risk

In the earlier part of this century -- those halcyon days before the bottom fell out of the global economy -- I met a guy named Skip Snow whose job at Citigroup was, as he told me, to "think about governance" for SOA. At the time I remember thinking, man, I'd like to have his job.

To my disappointment, Skip explained to me recently that his position as senior vice president of enterprise architecture for Citicorp (aka, "chief architect for SOA") was less pleasant than sitting in the backyard hammock staring at clouds. I called Skip at the offices of his startup, HIPAA Box, a company he founded to offer secure preservation of data held by health care providers.

[ Early on in the crisis, InfoWorld blogger Bill Snyder wondered: Will the financial meltdown slow IT innovation? ]

At Citicorp, he was fighting the good fight for SOA -- and losing the war. If he and architects at other financial giants had won, says Skip, the meltdown might never have occurred. How can he make such a fantastic claim? Because, he says, SOA would have made massive financial risk across the organization impossible to ignore. And as usual, political rather than technical barriers stood in the way of essential change.

"What we had in the 2003 timeframe was a bunch of guys leading the industry who knew what was required to make SOA work," he said, "but at Citicorp the internal competition didn't allow it."

Such internecine inertia was not exclusive to Citicorp. To deploy an SOA successfully, you need to tease out common-denominator services across the enterprise and have applications share them to avoid duplicate effort and to centralize management. The architects saw the benefit, but what did those within each department have to gain? They were making money hand over fist already and didn't need the distraction. Besides, says Skip, "business unit A and business unit B wanted to maintain their fiefdoms. In fact, compensation models for business managers did not reward enterprise players."

Enterprise software vendors were complicit too, he says. "If you're an IBM or an Oracle or an HP, where's the benefit in having a commodified messaging infrastructure? It wasn't in their interest."

Instead, the big vendors want to lock you into a proprietary stack, even though it's impossible to adopt the same homogeneous stack across a vast organization like Citicorp. So you end up where you started: multiple stacks that communicate with each other only to a limited degree.

That lack of integration was ironic. The Gramm-Leach-Bliley Act of 1999, which removed the Depression-era prohibition against financial institutions conducting banking, investment banking, and insurance services under one roof, often takes the rap for enabling the mortgage-backed securities disaster that brought the world financial system to its knees. In truth, as huge companies like Citicorp raced to embrace this profitable triple threat, they never really integrated their operations, so the supersized risks of their new, supersized business avoided radar detection.

"The financial giants were particularly vulnerable," says Skip, "because their scale made it impossible to understand their own risk. I'm not saying that greed was not apparent. But had those kinds of black-and-white risks been irrefutable, things might have gone very differently."

According to Skip, Citicorp had multiple, nonintegrated risk management systems and multiple general ledgers that did not roll up neatly to one reporting engine at any level of detail. It was easy to ignore risk at the macro level because the bank's portfolio could not be aggregated into a single data federation or warehouse. Skip speculates that if company executives had recognized their exposure to a fast decline in the value of retail mortgages, they might have shed some of those assets prior to the meltdown -- or at the least, Citigroup's contribution to the meltdown would have been significantly smaller.

According to Skip, a number of architects saw the risk, very likely before anyone else did. "Maybe architects are supposed to be the canary in the coal mine. It's their job to solve the impedance mismatch between business and technology. Many knew that, hey, this isn't as good as it should be."

It's part of an architect's job to talk to all kinds of people who administer business-critical systems. In January, Skip got advance warning into how bad the situation at Citicorp had gotten. "I was talking to a guy in the commercial bank and he told me point blank: 'We have not sold a mortgage since August.' Car loans were down by half. I knew then this much bigger than I'd realized." The writing was on the wall. Skip got out before the mass layoffs.

Today, Skip has made the transition from SOA advocate to cloud service provider, a business idea inspired by the health care IT challenges he encountered as vice president for technology architecture at Kaiser Permanente. HIPAA Box tackles a complex problem: backing up medical data for providers of all sizes, from clinics to hospital networks, while maintaining strong security and privacy protection. Under the hood, HIPAA Box is pretty sophisticated stuff, with agents that run on customer servers to enable a rule-based, federated backup that links distributed data and keeps it coherent.

The value proposition for health care providers, though, is simple: peace of mind that -- in relation to their data, at least -- HIPAA compliance is under control.

In the future, let's all hope the financial services industry seeks a similar sense of equanimity rather than blindly inflating another bubble. If they have any questions about that, they should talk to their architects.