TRW Automotive, a Tier One global supplier of auto safety products, had a big control problem. “We had always operated as a dozen or so largely independent business units, each with its own systems,” says Joe Drouin, vice president and CIO. “As a result, we had dozens of mission-critical applications, including multiple flavors of SAP, running in dozens of different places.”
Compliance and disaster recovery were particularly challenging, Drouin recalls. “In our first year of Sarbanes-Oxley, far too much time was spent on internal and external audits of every server and application that was anywhere. We also had one of those classic disaster-recovery plans that was really just a plan, and gosh help us if we ever had to implement it.”
The solution: a major consolidation effort. TRW Automotive went from dozens of distributed datacenters to four. And centralized Dell/EMC SAN storage made a viable disaster-recovery strategy much more practical and affordable, while helping the company meet its compliance responsibilities.
Consolidation efforts such as TRW’s reflect the imperative that has ruled IT for the past five years: Do more with less. On the ground, that means centralizing and rationalizing datacenters, servers, applications, management tools, and network infrastructure -- the result of mergers, acquisitions, and unconstrained technology growth -- into fewer, standardized, more manageable datacenters and platforms.
Consolidation recently has been given an extra boost, thanks to new technologies such as virtualization and 64-bit, multicore-core hardware that packs more punch into a smaller footprint. At the same time, expectations have risen sharply.
Reducing Costs, Raising the Bar
Five years ago, the main motivation for consolidation was cost reduction. “After the Internet bubble burst, lots of companies looked for ways to cut 30 percent or more from their IT budgets,” says Steve Fink, who runs HP’s IT consolidation group. “Many succeeded through consolidation. Today, however, it’s all about synchronizing business and technology to be more responsive to the customer than your competitors.”
Consolidation can be an effective way to gain that agility. “Companies have found that consolidation strategies such as [server and storage] virtualization not only allow them to do more with the resources they have, but to be more flexible, agile, responsive, and available as well,” says Michelle Bailey, vice president at IDC. For example, using server and storage virtualization, provisioning can be completed in a fraction of the time.
But agility and efficiency aren’t the only motivators. For First American, a leading financial information provider, high availability was key. “As the company became a major information provider, we knew we had to provide super reliability in a very modern datacenter,” says First American CTO Evan Jafa. First American has so far consolidated 25 locations to two datacenters, which it built from scratch to provide high-density power and cooling.
The company is not alone. “With the rise of high-density equipment such as server blades, half the datacenters out there are on the way to becoming functionally obsolete within two to three years,” says Michael Bell, research vice president at Gartner.
Centralized business process automation was the motivation for First American Title Insurance, part of First American. “We had 50-plus title and escrow production systems on every single variation of hardware and operating system,” says Larry Godec, CIO of First American Title Insurance. “We were looking to centralize, re-engineer, and automate all that back-office work.”
Discover, Analyze, Strategize
Lofty goals are great, but enterprises also need to be aware of risks. Consolidation is a potential logistical nightmare that, if poorly managed, can take far too long and be far too disruptive to both IT and the business.
One of the first steps in an effective consolidation is to define what is to be gained and how that serves the long-term goals of the organization. Is the first objective cost savings, agility, process transformation, disaster recovery, or compliance? Agreement, which should be explicit, requires ongoing communications with upper management and all affected business units.
The next step is to discover and analyze all the affected systems, storage, and applications across the network, understand their interdependencies and staff and their resource requirements, and use that data to build a business case and strategy for consolidation, based on a specific set of objectives.
“Being a statistical agency that was good at surveys, we did a survey in the departments to see what hardware, storage, and backup was out there, and tried to associate full-time employee costs to each of those environments,” says Guy Charron, assistant director of infrastructure services at Statistics Canada, the Canadian government’s statistics-gathering agency. “We also looked at the role each server played in the environment.”
Such fact finding is essential, Gartner’s Bell says. “You need to develop a complete inventory of what you have, what is near the end of life, what you want to retire or replace, what new equipment should you add,” he says. “You also need to decide if you want to maintain a regional network of datacenters, due to some unique requirements of each region, or just move to one or two primary datacenters and backup facilities.”
Charron credits Statistics Canada’s consolidation success to getting all the parties involved at an early stage. “Each division had representatives participating in building the business case and agreeing on the numbers,” he says. The survey and business case took four to five months.
HP’s Fink puts the psychology succinctly: “Everyone has to think it was their idea and have the metrics to show their boss all that money they saved.”
Even with broad involvement, however, there’s often resistance, particularly when it comes to moving servers and applications to a central location. “Everyone wants to hug their servers before they go home,” Charron says.
There’s also fear of major disruption, particularly in the case of application consolidation. “We decided to consolidate at the height of the refinance boom, when people were very busy,” First American Title’s Godec says. “The idea went over like a lead balloon.” First American Title’s experienced escrow officers had prided themselves on their efficiency in getting documents produced in a timely manner. “Now they would no longer do that work. I think they felt outsourced.”
Resistance escalated during the rollout. “They were screaming bloody murder.” It didn’t help that the more experienced employees were having a harder time adjusting to the new system than those on staff for two weeks, who were more productive sooner. “Two weeks later, though, they were all thrilled,” says Godec, who adds that some quit but many have come back. Godec emphasizes that a big success factor in application consolidation is focusing heavily on the human element. “Make sure you overtrain and are right there with the end-users during the change.”
Resistance can also come from IT, but it’s often not as serious as one might expect. “There was the usual fear of job loss, and we worried that we’d lose key skills too early,” says John Carrow, vice president of strategic client development and former CIO at Unisys, which has been consolidating in waves since the late ’90s. “We thought about retention programs, but it turned out we didn’t need them. The techie work culture got caught up in the technology and understood the learning opportunities it created.”
Nonetheless, consolidating soon after a company merger can exacerbate tensions, as it did at Advantage Sales and Marketing, a leading marketer of consumer packaged goods. “It was really tough,” says William Hiatt, the company’s national technology director. “We were the big, bad evil corporation coming in and changing everything. A lot of IT left so we had to hire new people even while we were cutting to reduce the budget.”
Many companies make a concerted attempt to redeploy IT workers. At KMD, a Danish software developer for local governments that used VMware’s ESX Server to consolidate three datacenters to one, many of the same staff continued managing servers and applications remotely in virtual teams after the consolidation. “We needed to make sure each of the three departments was well represented on the technical side,” says Alan Madsen, a systems engineer at KMD.
But cuts are often inevitable. “If you’re going from 5,000 servers to 2,500, you’re obviously not going to need all those system administrators,” IDC’s Bailey says.
Taking Baby Steps
Yet TRW left PeopleSoft running on HP-UX. “We had thousands of users running those applications,” Drouin says. “Changing those platforms in the middle of a transition was just too big a bite for that first stage.”
Advantage Sales and Marketing’s first step was messaging. “E-mail was the low-hanging fruit,” Hiatt says. “It was so antiquated, so we started by building a unified messaging system and active directory domain, which was well received. Then we started focusing on virtualization and consolidating database servers.”
Unisys started by consolidating regionally in the late ’90s. The second wave was to consolidate to a single global datacenter in Minnesota. Only then did the company start focusing on applications, such as consolidating to Oracle ERP for financials and Siebel for CRM. Even then, it retained two instances, one for the U.S. and one for international users. “As it turned out, it was a smart move because we need the redundancy for business continuity,” Unisys’ Carrow says. Finally, the company started consolidating messaging under Exchange.
“It’s one thing to talk about consolidating hardware and another to talk about databases and applications, which are much more complex and more visible,” IDC’s Bailey says. “If you take on too much, you get cost overruns and a lot of pressure to just get it out, so the project ends up falling short.”
AT&T’s reputation rests so heavily on faultless reliability, that it often starts consolidating by simply replicating all existing systems at the new location. “Collapsing the footprint is much more easily achievable than collapsing servers and applications, and it has a big return on investment in terms of management, efficiency, and real estate savings,” says Dave Roberts, senior technical director of consolidation, planning, and relocation at AT&T.
First American’s Jafa advises not to get too obsessed with standardization for standardization’s sake. “You want to serve the customer need for things like scalability and reliability, not force them to suddenly change their cost model or negatively impact them with a limited selection of standards.”
Plan ... Then Plan Some More
After the strategy has been decided, execution requires very careful, detailed planning. “You really need to discover all the details, all the relationships among systems and networks that you’re going to move around, and do lots of planning for everything that might happen,” Jafa says. “And you need to really focus on the lessons learned with each consolidation and use them to develop standard approaches, processes, and implementation plans for the next one.”
Business units should also be involved in the logistics planning. “We’d turn over many of the plans to the business units. The ultimate go/no-go decision was always made by the business, not by IT,” Jafa says. Similarly the business units were the ones that decided whether things were functioning the way they should.
Because it had small windows for each actual move, TRW built in specific drop-dead milestones. “If it got to be Saturday at 10 p.m. and we weren’t where we expected to be, we had to roll everything back, no questions asked,” says Anil Goindi, senior IS manager at TRW. Dry runs in a test environment were also essential.
For maximum reliability, AT&T replicates data to the target location and updates both systems in parallel for a full six months before going live at the new location. “We’ve learned that data movement is absolutely key to success,” Roberts says.
Getting help from vendors is also a big ingredient for success. “You have to tag along with the tech experts for each system,” First American’s Jafa says. “We got a lot of help and used a lot of best practices from our primary partners Fujitsu, HP, and IBM.”
Consolidation is hard, and some disruption is inevitable, but when it’s done right, the benefits are real. First American Title estimates $5 million in annual IT savings, with a total of $100 million in annual savings and revenue due to accelerated business processes. “We’ve integrated our lenders’ systems into our centralized system so we can have all the lenders’ instructions in before closing. Before we had to kill time feeding the family cookies in the waiting room while the escrow officer was still scrambling to acquire, key, and print the most up-to-date, final information. None of our competitors can do this.”
Statistics Canada estimates savings of $2.9 million in the first four years, and despite massive storage growth from a recent census, it has had to hire only one more full-time employee. Unisys estimates shaving $50 million out of the run-rate cost of IT operations, while TRW and First American were able to use saved space and decommissioned servers to build a first-rate disaster-recovery infrastructure.
Perhaps the most dramatic benefit, however, is agility. “After we realized how much more agile we were, and how successfully we had made all the changes, we became much more creative and confident about what we could do,” TRW’s Drouin says.