If you're reading this column, the odds are about even that you've been impacted in some way by the OpenSSL Heartbleed vulnerability. And if you know someone who hasn't heard of it, tell them immediately because everyone in the known world should change all of their passwords to everything, apparently.
For the past week we've been deluged with news about the problem and a steady outpouring of information, tips, methods, and solutions to address the issue. These were definitely not the days to be responsible for Web hosting infrastructures or anything else to do with SSL security.
[ The InfoWorld Test Center review: Puppet vs. Chef vs. Ansible vs. Salt | The joy of data center automation -- and its hidden dangers | Get the latest on data center hardware and software with InfoWorld's Data Center newsletter. ]
There are many takeaways and lessons to be learned from this experience, but I'll go through two that struck me again and again this week. The first is that if there ever was an open-and-shut case to be made for data center orchestration and centralized server management, this was it. Second, it pays to be a bit behind the curve on production systems.
I've read more than a few synopses of how large infrastructures were handling this announcement. It's one thing to have to find and patch a few dozen Web servers and gold server images, but it's quite another to have to deal with thousands of active and vulnerable Web hosting servers.
That's where data center orchestration comes into play -- at least for part of it. Those who already had robust and tested management tools like Salt, Ansible, Puppet, or Chef reaped the rewards this week. This wasn't just an administractive timesaver, but the siren song of rapid and stable configuration management frameworks. This was what let you distribute patched OpenSSL packages to potentially thousands of servers, virtual and physical, and restart the right daemons within hours, if not sooner. In the scope of the Heartbleed situation specifically, this was absolutely huge.
Let me take a small step back and underline that point. The Heartbleed vulnerability allowed malicious code to view snippets of active memory from an SSL-enabled process. This meant that an attacker could siphon small bits of active memory from a Web process without any trace of the attempt being logged or otherwise noted unless the traffic was collected by a packet sniffer. Now, the attacker couldn't specify what data they wanted out of the process memory, but they could continue to siphon 64KB chunks of memory as often as they liked, until they found what they were looking for: the private key, user names, passwords, file data, you name it.
The longer a vulnerable server answered requests, the more likely a bad actor was able to access usable, sensitive data. The speed to close that hole was exceedingly important with Heartbleed.
With that in mind, if it took less than an hour to put together an orchestration job to update a few OpenSSL packages and restart a few services on hundreds or thousands of servers, as opposed to several hours or several days, you were far less likely to lose sensitive data or have your keys picked. With Puppet, Chef, Salt, or Ansible, beating that hour window was a very achievable outcome. This cannot be overstated.