Sometimes a simple piece of hardware or software -- and the presence of mind to install it -- can make all the difference in the world when the walls come crashing down. One of these items is the oft-ignored serial terminal server. In fact, I'd wager that many IT professionals don't even know what one is offhand.
I genuinely miss the Cisco 2511 terminal server. Outfitted with a couple of octopus cables, it was a trusty Swiss Army Knife during the golden age of the Internet.
[ As this networking Deep Dive attests, InfoWorld contributing editor Paul Venezia is a veritable font of network wisdom | Also see Matt Prigge and Paul Venezia's "10 tips for boosting network performance." ]
I've used 2511s to run modems in the days when 33.6kbps was warp speed. I've used them as routers, and I've used them as, well, terminal servers and connected those cables to the serial consoles of other network devices, servers, and whatnot. They've long since gone end-of-life -- but there are plenty of alternatives on the market. In fact, you can pick up used 16-port serial terminal servers from Avocent (who bought Cyclades four years ago) or Perle for as little as a few hundred dollars, depending on port count.
That $200 can easily be the difference between a ruined weekend and a nonissue. Sometimes, it can turn a potentially a massive and prolonged outage into a blip on the radar. And terminal servers aren't just for network devices.
In many cases, you can configure server hardware to ignore the framebuffer and output straight to serial. As long as you're not running a graphical operating system, you can do everything you want to a server from a standard serial connection, from BIOS modifications to rescue booting to full-on administration. If you're running a GUI, you can at least get to the BIOS out of the box.
A serial console connection is also generally faster than KVM-over-IP or an ILO remote console session. It's also much more robust over high-latency and low-bandwidth links, which means you're more likely to be able to fix a problem from your cell phone over a serial console session than via any other method.
All network hardware and a wide variety of other devices like storage arrays, blade chassis, UPS systems, and even air conditioning units have a serial console. In most cases, it's as simple as plugging in a cable -- voila, instant out-of-band management access to that device as long as it has power.