We've all been there. It's midnight, the maintenance window for work on a few remote systems is quickly closing, and some critical infrastructure component is down hard. It might be a router, a switch, a storage array, or a server -- it doesn't really matter. There's trouble in River City.
To the casual observer, it might seem that the hardest part of this scenario is finding and fixing the actual problem causing the outage. But in many cases, that's the least of it. The hard part is trying to do something absurdly simple -- such as properly generating a break sequence during the boot cycle.
A good example of this is a recent situation I fell into when a Sun Oracle storage array was throwing critical errors on boot following an update. The serial console was rife with error messages and prompts for further action. Each option was in the form of ESC-2, ESC-4, and so on. That was all well and good when I was SSH'd into the serial console server and driving the console directly, but when I connected to the service processor directly through SSH using the Sun Oracle Java-based shared shell applet, there was no way to generate the required Escape key. I wound up switching the console between the shared SSH session and my serial connection just to hit the Escape key. It wasn't just for this function -- try using vi without a functional Escape key.
That's only a minor example. I can recall spending the better part of 30 minutes trying to get into the BIOS of a server that had a criminally short wait for a key combination during the POST. Working through an IP KVM to a remote site with a bunch of lag and lacking the ability to generate the key combination natively, I had to use the KVM's macro functions to try and hit it just right, boot after boot after boot. Once I got in, fixing the actual problem on that box took only about 30 seconds.
I know we've all dealt with mystery mouse cursor motions with remote consoles. I recall a cheer going up from onlooking admins when a spastic cursor finally, miraculously landed on the right button in a dialog box on a server that was 2,000 miles away -- after at least 25 minutes of trying. And no, there was no corresponding hotkey for that function. Situations like that really make you want to hunt down the developers responsible and whack them with a sack of doorknobs.