I've written before about the huge benefits you can reap if you plan for large sitewide outages by giving yourself access to all the troubleshooting tools you'll need ahead of time. These days, that almost always includes access to a bevy of embedded management interfaces. These interfaces are common on devices like uninterruptible power supplies, network-attached power distribution units, blade chassis, and server hardware in the form of baseboard management controllers (BMCs). They can be an enormous help when you've had a full site failure or are remotely troubleshooting a huge range of problems.
However, they also can present an enormous risk if not protected properly.
Recently, US-CERT released a security advisory that explains the risks inherent with exposing Intelligent Platform Management Interface (IPMI) interfaces to unsecured networks. IPMI is an API standard maintained by Intel that describes a platform-independent method of interacting with the BMCs on servers. This advisory followed the release of numerous vulnerabilities in the IPMI 1.5 and 2.0 standards discovered by independent security consultant Dan Farmer while working on a DARPA grant.
Effectively, the vulnerabilities Farmer discovered allow unfettered access to the most basic functions of any server with an exposed and unpatched IPMI interface. Given that more than 200 server manufacturers have adopted Intel's standard (Hewlett-Packard, Dell, SuperMicro, IBM, you name it) and implemented it in their own BMCs, chances are your data center is full of enabled and accessible IPMI devices. Effectively, a hacker with full access to an IPMI interface might as well be physically sitting in front of the server -- that's the point of these interfaces.
Although Farmer's research centers on the widely used IPMI standard, the idea that a shadow set of black-box servers -- many using other protocols -- operate on our networks and run old, infrequently patched code over which we have very little control is extremely disturbing. You don't have to think very hard to come up with more devices just like this: printers, UPSes, power distribution hardware, network-attached storage appliances, time clocks, search appliances, you name it. We all run a lot of "invisible" servers on our networks -- sometimes without realizing it.
There's very little you can do to actually secure these kinds of devices. Certainly, it's helpful to keep an ear to the ground for advisories and patch devices when a vulnerability is discovered, but many times that notification will simply come too late. All too often, the manufacturers of these embedded devices simply don't care enough to keep them up to date with patches.
If you need a great example, see Roger Grimes' piece on white-hat hacking -- specifically the first story in which he and a group of security analysts perform penetration testing on a set-top IPTV box. In his story, the cable company had at least hired someone to double-check the security on a device that it was about to deploy (a rarity), but in the process of the work, it found a vulnerability that existed on hundreds of thousands of other, already deployed boxes.
At the end of the day, the only thing you can reliably do to prevent these devices from being misused is to simply shut them off or disconnect them -- nothing is more secure than a network device that isn't plugged in. However, in the case of BMCs and IPMI, even that simple fix can be a challenge. Some BMCs share one of the server's onboard NICs by effectively implementing an internal network switch. Thus, they're impossible to physically disconnect, although you can usually disable them completely in the system's BIOS.
Even if you could physically disconnect them all, you don't want to. There's a reason you paid money for server hardware and network devices that have this kind of management capability: You want to make your life easier and decrease downtime in disasters. You can't use this hardware when you need it if it's not connected.
The next best thing to not plugging them in in the first place is to segregate them from everything else on your network and lock access down as much as you can.
At the very least, you need to first identify all the potentially vulnerable network devices on your network and corral them into an isolated LAN (either a VLAN or one operated by physically separate switching). Next comes the challenge of figuring out how you will access that "supersecure" LAN from your normal LAN (or remotely!) -- the last thing you want to do is give someone access to all your easiest targets gathered in one place.
Whatever you do to secure these kinds of embedded devices, the first challenge is to realize that they exist and can be a real threat to your infrastructure. Absolutely don't expose these kinds of devices to completely unsecured networks -- the Internet, for example. Consider either isolating them in their own secure network or disabling them entirely if you don't use them.
Given how widely Farmer's research has been disseminated, you can bet that worms are already being developed to take advantage of the vulnerabilities he discovered -- so don't waste any time.
This article, "Those 'invisible' servers could open your network to hackers," originally appeared at InfoWorld.com. Read more of Matt Prigge's Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.