Any network admin who's been around for a while has had this experience. It's not something you look forward to doing, nor is it arguably a good idea. You do it because it's really the only option at the time -- a remote network cutover that relies on the same access path used for the reconfiguration.
Picture it: a remote location, starved for bandwidth, maybe 200 miles away from anyone who can lay hands on the gear. A new egress circuit has just been installed to bring network access into the 21st century. The options are to perform this task remotely or to have someone drive four hours each way.
[ Want more networking wisdom? Download the Networking Deep Dive Report by Paul Venezia. | Read Paul Venezia's post on remote monitoring and control systems, "Stay connected when disaster strikes." | Get the latest practical info and news with InfoWorld's Data Center newsletter. ]
The decision process usually involves several people who don't want to drive and a network admin who can do the work remotely but cannot guarantee its success. At some point, the realization hits: If the worst-case scenario is a late-night drive if the reconfiguration goes south, then why not try to do it remotely and send someone out if it fails?
Thus begins the game. Every possible method of gaining access to the gear from a different network path is inspected and ultimately discarded. It might be nice to have a modem hanging off a box for this remote access, but there's no good way to do that. Remoting in via cell or other wireless service is also a nonstarter. This will have to be done the hard way.
The admin pulls down configs for the firewall, routers, and associated switches, then freehands a new configuration that will remap the interfaces properly (he hopes), set new routes (he hopes), and change VLANs here and there. It's a strict order-of-operation task, as jumping ahead one step will spell disaster and fat-fingering a single command will cut off every avenue of access, rendering a remote fix impossible. The admin writes all the existing configurations to the devices so that in the event of disaster, they can be power-cycled to return to their previous state -- though that'll be small consolation to the night driver.
After verifying everything he can, double-checking every single statement, and setting up tests to verify the cutover, he closes his eyes, cracks his knuckles, and carefully begins the process. The clock hits 11 p.m.