Patch management, even on a departmental scale, can be one of the toughest challenges facing IT. Factor in a geographically
dispersed enterprise, and the job becomes absolutely daunting.
|
2006 InfoWorld 100 Winners
|
|
|
|
Consider the situation at the Internal Revenue Service. The IRS’s administrative network -- separate from its tax processing
systems -- encompasses some 100,000 workstations in 400 offices nationwide, including Hawaii and Puerto Rico. In addition,
some employees work from home, while others are located on-site at businesses or in telecommuting centers. As you can imagine,
keeping on top of patches for an environment this complex isn’t easy.
“We did some checking and learned that actually most of our workstations have most of the patches,” says Allan Roberts, program
manager for enterprise systems management at IRS. “But when you have an environment of 100,000 workstations, if you miss an
average of just one or two or three per workstation, those numbers add up.”
In fact, Roberts estimates that the total was close to 1 million missing patches throughout IRS’s network. Not all of them
were critical, but if even a fraction of these represented exploitable security vulnerabilities, the situation could be serious.
“Once we began to grasp the scope, we felt that there was some urgency to it,” Roberts says.
The answer to the IRS’s patch management woes was a combination of off-the-shelf applications and home-grown solutions. The
IRS uses IBM Tivoli Configuration Manager to automate distribution of packages throughout its administrative network, but
that software alone couldn’t guarantee successful completion of its patch cycles.
“We’ve been using [Tivoli], and we’ve had great success with it, [but] there’s been enormous support costs that go with it,
too, to make it work within our environment and the way that we operate,” Roberts says.
Tina Walters, the IRS’s program manager for workstation standards, says that any out-of-box software just couldn’t meet its
demands without heavy support and customization.
“When you look at our environment and how complex it is and how large it is, normally when we go out and look at vendor products
to meet one of our needs, it takes sometimes more resources to maintain and manage that product because we have to tweak it
and refine it to meet our needs -- because of our regulations, our rules, our requirements,” Walters says.
So Walters and her team bridged the gap with a custom solution consisting of scripts that could automatically audit a workstation
for currently applied patches on a routine basis and compare the results with a master list of available patches, hosted on
an internal Web server. Whenever discrepancies are found between the preferred configuration and the patches installed on
the workstation, the system can trigger other scripts to download and install the missing fixes automatically.
“It’s all distributed electronically,” Walters says. “No one has to go to the workstation and touch it.”
The IRS is now that much closer to its ambitious patch maintenance goals. The system isn’t perfect, Roberts says, but if a
workstation does lose connectivity to the IRS network for a period of time, it doesn’t mean it goes without crucial software
security updates.
“The last I checked, we had passed three-quarters of a million patches that we had caught up,” Roberts says. “And the meter’s
still running. So it has just been an extraordinary success story for us, to reduce the vulnerabilities on our network.”
2006 INFOWORLD 100 WINNERS
Top 10 Finalists
Computing/Tech Distribution/Supply Chain
Education Financial Services
Government Health Care
Insurance Manufacturing
Media/Multimedia Pharmaceuticals
Retail Services
Sports Telecommunications
Transportation Unspecified