Patch management, even on a departmental scale, can be one of the toughest challenges facing IT. Factor in a geographically dispersed enterprise, and the job becomes absolutely daunting.
Consider the situation at the Internal Revenue Service. The IRS’s administrative network -- separate from its tax processing systems -- encompasses some 100,000 workstations in 400 offices nationwide, including Hawaii and Puerto Rico. In addition, some employees work from home, while others are located on-site at businesses or in telecommuting centers. As you can imagine, keeping on top of patches for an environment this complex isn’t easy.
“We did some checking and learned that actually most of our workstations have most of the patches,” says Allan Roberts, program manager for enterprise systems management at IRS. “But when you have an environment of 100,000 workstations, if you miss an average of just one or two or three per workstation, those numbers add up.”
In fact, Roberts estimates that the total was close to 1 million missing patches throughout IRS’s network. Not all of them were critical, but if even a fraction of these represented exploitable security vulnerabilities, the situation could be serious.
“Once we began to grasp the scope, we felt that there was some urgency to it,” Roberts says.
The answer to the IRS’s patch management woes was a combination of off-the-shelf applications and home-grown solutions. The IRS uses IBM Tivoli Configuration Manager to automate distribution of packages throughout its administrative network, but that software alone couldn’t guarantee successful completion of its patch cycles.
“We’ve been using [Tivoli], and we’ve had great success with it, [but] there’s been enormous support costs that go with it, too, to make it work within our environment and the way that we operate,” Roberts says.
Tina Walters, the IRS’s program manager for workstation standards, says that any out-of-box software just couldn’t meet its demands without heavy support and customization.
“When you look at our environment and how complex it is and how large it is, normally when we go out and look at vendor products to meet one of our needs, it takes sometimes more resources to maintain and manage that product because we have to tweak it and refine it to meet our needs -- because of our regulations, our rules, our requirements,” Walters says.
So Walters and her team bridged the gap with a custom solution consisting of scripts that could automatically audit a workstation for currently applied patches on a routine basis and compare the results with a master list of available patches, hosted on an internal Web server. Whenever discrepancies are found between the preferred configuration and the patches installed on the workstation, the system can trigger other scripts to download and install the missing fixes automatically.
“It’s all distributed electronically,” Walters says. “No one has to go to the workstation and touch it.”
The IRS is now that much closer to its ambitious patch maintenance goals. The system isn’t perfect, Roberts says, but if a workstation does lose connectivity to the IRS network for a period of time, it doesn’t mean it goes without crucial software security updates.
“The last I checked, we had passed three-quarters of a million patches that we had caught up,” Roberts says. “And the meter’s still running. So it has just been an extraordinary success story for us, to reduce the vulnerabilities on our network.”
2006 INFOWORLD 100 WINNERS
Top 10 Finalists
Computing/Tech Distribution/Supply Chain
Education Financial Services
Government Health Care