Brain drain: Where Cobol systems go from here

When the last Cobol programmers walk out the door, 50 years of business processes encapsulated in the software they created may follow

David Brown is worried. As managing director of the IT transformation group at Bank of New York Mellon, he is responsible for the health and welfare of 112,500 Cobol programs -- 343 million lines of code -- that run core banking and other operations. But many of the people who built that code base, some of which goes back to the early days of Cobol in the 1960s, will be retiring over the next several years.

"We have people we will be losing who have a lot of business knowledge. That scares me," Brown says. He's concerned about finding new Cobol programmers, who are expected to be in short supply in the next five to 10 years. But what really keeps him up at night is the thought that he may not be able to transfer the deep understanding of the business logic embedded within the bank's programs before it walks out the door with the employees who are retiring.

[ Subscribe to InfoWorld's Developer World newsletter for more on software development. ]

More than 50 years after Cobol came on the scene, the language is alive and well in the world's largest corporations, where it excels at executing large-scale batch and transaction processing operations on mainframes. The language is known for its scalability, performance and mathematical accuracy. But as the Boomer generation prepares to check out of the workforce, IT executives are taking a fresh look at their options.

In a Computerworld survey of 357 IT professionals conducted recently, 46 percent said they are already noticing a Cobol programmer shortage in the market, while 50 percent said the average age of their Cobol staff is 45 or older; 22 percent said the age is 55 or older.

"Organizations are trying not to get backed into a corner because of the skills issue," says Paul Vallely, mainframe sales director at software vendor Compuware Corp. "I haven't seen companies move off mainframes due to the Cobol skills shortage yet, but it's looming."

For Bank of New York (BNY) Mellon, which bought its first mainframe in 1955, keeping the core Cobol applications that run the business on the mainframe makes sense. Modernization efforts have made those programs more accessible through the use of Web services and more up-to-date user interfaces.

But for other types of non-core applications, and for smaller workloads, organizations have been gradually migrating off of mainframes -- and away from Cobol. In some cases those Cobol programs are simply re-hosted on Linux or Windows servers; in other cases they're rewritten in other, object oriented languages; and some programs are being replaced with packaged software.

"Over the past five years there has been an acceleration of [some] businesses moving off host platforms," says Adam Burden, global application modernization lead at Accenture. Often that means leaving Cobol behind by either rewriting it for J2EE or .Net or moving to packaged software.

Over the last few years, Gartner estimates that the world has seen about a 5 percent decline in total Cobol code, says analyst Dale Vecchio. Much of that involved migrations by small- and medium-sized mainframe shops that move off what they see as a legacy language when they retire the hardware.

It's declining because the functions can be developed by some other building block. "Cobol is no longer needed," Vecchio says. "There are alternatives."

While rehosting can get code off the mainframe quickly -- Micro Focus sells a platform that will support it on a Microsoft Azure cloud -- that is often seen as intermediate step. It can be used as a way to get Cobol off the mainframe quickly, before eventually completely modernizing and transforming those legacy programs.

Cobol's image problem

Cobol, as a procedural language, is not perceived to be as agile as object-oriented languages for modern programming needs such as mobile apps and the Web. And despite the availability of state-of-the-art Cobol development environments -- including IBM's Enterprise Cobol on the mainframe and Micro Focus' Visual Cobol, which integrates well with Microsoft's Visual Studio development suite for .Net -- Cobol is widely viewed as a legacy language.

Nearly half (49 percent) of survey respondents whose organizations don't use Cobol say the reason is that the language is simply outdated.

Not everyone agrees, of course. "Cobol has had lasting value, and it's not broken," says Kevin Stoodley, IBM Fellow and chief technology officer of enterprise modernization tools, compilers and security at IBM.

Some 64 percent of Computerworld readers concur, and say their organizations still use Cobol -- more than any modern language except for Java/JavaScript and Visual Basic. The last time we ran a Cobol survey, back in 2006, some 62 percent of respondents said they still used Cobol at that point.

In the more recent survey, over 50 percent of respondents say Cobol represents more than half of all internal business application code.

"There has been no renaissance for Cobol," says Accenture's Burden. "There's not a whole lot of new development going on. But our clients are enhancing their core applications and continue to maintain them." Indeed, 53 percent of readers say they're still building at least some new business applications in Cobol. The vast majority of that code is still being written for the mainframe.

But IT organizations also don't have much choice. Migrating large-scale systems built in Cobol is just too costly and risky a proposition. "They might want something more flexible but they just can't do it. They're captive to Cobol," Burden says.

The down economy has helped put off the inevitable, Compuware's Vallely says. "Economic issues provided everyone with a hall pass because not as many folks were looking to retire," he says. But as the economy improves, retirement plans may pick up too. "Organizations are trying to be more proactive," he predicts.

"No other language has seen as big an impact from changes in the demographics of the workforce as has Cobol," Vecchio says. Going forward it will become more difficult to maintain a Cobol portfolio.

"The inflexion point will come when enough Cobol programmers have retired that an organization can no longer tolerate the risk," he says. At that point, most of those programs will migrate -- but not all.

Rightsizing Cobol

For BNY Mellon, those Cobol batch and transaction processing programs on the mainframe represent an enormous investment. And while Gartner says it's technically possible to move mainframe workloads of up to 3,000 MIPS, the workload at the bank, which relies heavily on Cobol, consumes 52,000 MIPS of processing horsepower, spans nine mainframes and is growing at a 10 percent clip each year.

"The business wants us to make investments in programming that buys them new revenue. Rewriting an application doesn't buy them any value-add," Brown says.

Instead, the strategy is to "rightsize" some non-core applications off the mainframe where there's a business benefit, try to keep mainframe MIPS growth under 5 percent, and stay the course with the bank's core Cobol applications by passing on the business knowledge to younger programmers the bank will need to recruit and train.

Other functions, such as general ledger and reporting, are moving onto distributed computing platforms, where they are either replaced by packaged software or reengineered into Java or .Net applications.

But Brown still needs Cobol programmers to replace those expected to retire, and the learning curve can extend out for a year or more. That means adding staff and having a period of overlap as Cobol's secrets get passed on to the next generation. "I'm trying to get those people onboard and do the knowledge transfer sooner rather than later," Brown says.

But that kind of proactive approach, and the extra costs it incurs, can be a hard sell. "We haven't gotten to the point of feeling the pain yet. When we do, it will happen," he says.

Brown wouldn't specify the number of people he's hoping to hire, but said that the "real heavy need" will happen in the next five to 10 years, as the original mainframe programmers are expected to retire en force during that period. BNY Mellon currently has "a few hundred" Cobol programmers on staff, Brown says.

Brown's concerns are well placed, says David Garza, president and CEO of Trinity Millennium Group, a software engineering firm that has handled code transformations for large businesses and government organizations. "Almost every job we get has Cobol in it," he says, and most of the calls come from organizations that have already lost their collective knowledge of the business logic. At that point, he says, "it's a big risk."

The cost of waiting

Trinity Millennium Group and other vendors like it have established processes for analyzing and extracting the business rules embedded between the lines of Cobol code. "The solutions have come a long way in terms of the ability to extract logic and rules," says Burden.

But the process is time consuming and costly. One Millennium client recently spent $1 million to have its Cobol programs analyzed and business logic reconstructed as part of a migration project off the mainframe. "If they had the legacy programmers there and we had done the exercise with them, it would have cost $200,000 and taken one-tenth of the time," Garza says. If you wait until that institutional knowledge is gone, he warns, the costs can be as much as ten times higher than it would have been beforehand.

Compounding the loss of skills and business knowledge is the fact that, for some organizations, decades of changes have created a convoluted mess of spaghetti code that even the most experienced programmers can't figure out. "Some systems are snarled so badly that programmers aren't allowed to change the code at all," Garza says. "It's simply too risky to change it. They're frozen solid."

Package deal

That's the situation faced by Jim Gwinn, chief information officer for the USDA's Farm Service Agency. The USDA's System/36 and AS/400 systems run Cobol programs that process $25 billion in farm loans and programs. "We have millions of lines of Cobol and there's a long history of it being rewritten," he says. "It has become increasingly difficult to change the code because of the complexity and the attrition of the knowledge base that wrote it." That's a big problem because laws that govern farm programs change every year, driving a need to update the code to reflect those changes.

Gwinn hired consultants from IBM, who concluded that rewriting the programs in a different language or rehosting them on a distributed computing platform would be complicated and costly. But the System/36 hardware had to go, so Gwinn decided to bite the bullet: FSA will move off of its end-of-life mainframe systems by rewriting some of the code in Java and replacing the rest with packaged software from SAP.

But Gwinn says he'll miss Cobol. "It has been very stable and consistent, with little breakage due to code changes, which you see with Java-based changes," he says. "And in a distributed environment you have to balance your workloads a little more carefully."

Going for a rewrite

The anticipated exit of institutional knowledge and shortage of Cobol programmers were also primary drivers behind NYSE Euronext's decision to reengineer 1 million lines of Cobol on a mainframe that ran the stock exchange's post-trade systems. While Cobol was dependable, it wasn't viewed as maintainable in the long run.

Steven Hirsch, chief architect and chief data officer, cites the need to make changes very rapidly as another key reason the NYSE abandoned Cobol. "Ultimately, the code was not easily changeable in terms of what the business needed to move forward. We were pushing the envelope of what it took to scale the Cobol environment," he says.

So NYSE rewrote Cobol programs that run its post-trade systems for Ab Initio, a parallel-processing platform that runs under Linux on high-end Hewlett-Packard DL580 servers. The new environment allows for more rapid development, and the rewrite also eliminated a substantial amount of unnecessary and redundant code that had crept into the original Cobol programs over the years.

If the business' Cobol code doesn't need to change much, and many batch and transaction processing programs don't, it can be maintained on or off of the mainframe indefinitely. But that philosophy wouldn't work for NYSE. "We are a rapidly changing business and we needed to move faster than our legacy code," Hirsch says.

As for trading, that's all proprietary NYSE Euronext software. "There's no Big Iron or Cobol," Hirsch explains. "There's been no use of mainframes in the trading environment for many years."

Rehosting: Lift and shift

When it comes to hiring new Cobol programmers, Jonathan J. Miller, director of information systems and services for Saginaw County, Michigan, is struggling. "We've lost our systems programming staff," he says. And like many government IT organizations that have suffered from budget cuts, he doesn't have much to offer those in-demand Cobol programmers.

Generous government benefits used to attract candidates even when salaries were lower than in private business. Now, he says, "Our pay hasn't increased in eight years and benefits are diminished." To fill in the gap, the county has been forced to contract with retired employees and outsource Cobol maintenance and support to a third party -- something that just 18 percent of Computerworld readers say they are doing today.

The Cobol brain drain is starting to become critical for many government organizations, says Garza. "It's a high risk problem in many countries we are doing work in. The people have retired. Even the managers are gone. There's no one to talk to."

Saginaw County found itself hemmed in by the complexity of its Cobol infrastructure. It has four million lines of highly integrated Cobol programs that run everything from the prosecutor's office to payroll on a 46 MIPS Z9 series mainframe that is at end of life. With mainframe maintenance costs rising 10 percent to 20 percent each year, the county needs to get off the platform quickly.

But commercial software packages lack the level of integration users expect, and Miller's team doesn't have the time and resources to do a lot of integration work or reengineer all of the program code for another platform.

So the county is starting a multi-phase project to recompile the code with Micro Focus Visual Cobol and re-host it on Windows servers. An associated VSAM database will also be migrated to SQL Server. Miller hopes that the more modern graphical development suite will make the Cobol programming position, which has gone unfilled for two years, more attractive to prospective programmers. But he acknowledges that finding talent will still be an uphill battle.

A legacy continues

Is there a role for Cobol off the mainframe? "I don't believe there is. Cobol and the mainframe run well together, and that's where I want to keep it," BNY Mellon's Brown says. But the bank is still creating new Cobol components on the mainframe, and will continue to do so.

That's a common sentiment among Accenture's large corporate customers, says Burden. Cobol will continue its gradual decline as midrange systems are retired and businesses continue to modernize legacy Cobol code or move to packaged software. Today Cobol is no longer the strategic language on which the business builds new applications. But it still represents the "family jewels" of business, Burden says. "They're enhancing existing applications and adding functionality to them. I've seen no slowdown in those activities."

If companies can't find talent to keep that infrastructure going, outsourcing firms such as Accenture are ready, says Burden. The scale of Accenture's support operation is large enough to provide a career track for Cobol programmers, and he says it's easy to cross-train programmers on the language. "We can turn out new programmers quickly. So if clients can't support Cobol, we will."

"People make too much of that trend that we're not graduating enough Cobol programmers," says IBM's Stoodley. Preserving the institutional knowledge is what's critical. "You can make a problem for yourself if you don't keep your team vibrant," he says. But as long as there's a demand for it, "businesses will find people willing to work on Cobol."

"You have to respect the architecture of Cobol," Burden says. It may have been created for simpler times in application development, but it remains the bedrock of many IT infrastructures. "I don't see that changing for another 10 years, or even longer."

Robert L. Mitchell is a national correspondent for Computerworld. Follow him on Twitter at twitter.com/rmitch, or email him at rmitchell@computerworld.com.

Read more about Data Center in Computerworld's Data Center Topic Center.

This story, "Brain drain: Where Cobol systems go from here " was originally published by Computerworld.

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies