Mission impossible: A remote network cutover

Few feats are more rewarding to a network admin than pulling off the seemingly impossible -- which is not exactly a rare request

Any network admin who's been around for a while has had this experience. It's not something you look forward to doing, nor is it arguably a good idea. You do it because it's really the only option at the time -- a remote network cutover that relies on the same access path used for the reconfiguration.

Picture it: a remote location, starved for bandwidth, maybe 200 miles away from anyone who can lay hands on the gear. A new egress circuit has just been installed to bring network access into the 21st century. The options are to perform this task remotely or to have someone drive four hours each way.

[ Want more networking wisdom? Download the Networking Deep Dive Report by Paul Venezia. | Read Paul Venezia's post on remote monitoring and control systems, "Stay connected when disaster strikes." | Get the latest practical info and news with InfoWorld's Data Center newsletter. ]

The decision process usually involves several people who don't want to drive and a network admin who can do the work remotely but cannot guarantee its success. At some point, the realization hits: If the worst-case scenario is a late-night drive if the reconfiguration goes south, then why not try to do it remotely and send someone out if it fails?

Thus begins the game. Every possible method of gaining access to the gear from a different network path is inspected and ultimately discarded. It might be nice to have a modem hanging off a box for this remote access, but there's no good way to do that. Remoting in via cell or other wireless service is also a nonstarter. This will have to be done the hard way.

The admin pulls down configs for the firewall, routers, and associated switches, then freehands a new configuration that will remap the interfaces properly (he hopes), set new routes (he hopes), and change VLANs here and there. It's a strict order-of-operation task, as jumping ahead one step will spell disaster and fat-fingering a single command will cut off every avenue of access, rendering a remote fix impossible. The admin writes all the existing configurations to the devices so that in the event of disaster, they can be power-cycled to return to their previous state -- though that'll be small consolation to the night driver.

After verifying everything he can, double-checking every single statement, and setting up tests to verify the cutover, he closes his eyes, cracks his knuckles, and carefully begins the process. The clock hits 11 p.m.

This is not the time for distraction -- this is a time for laser focus on the task at hand. He can picture the pathways, the routes, each layer of the network as he ever so carefully peels them apart and makes the preliminary changes. Finally, he pulls the trigger and the devices pull in the freehanded changes, starting with the deepest device, which immediately goes offline, as it now requires pending upstream modifications to be accessible once more. 

On his screen, several terminal windows that had been showing healthy pings begin to display timeouts. Internal systems at the site blink offline. He switches from window to window, each logged into a different device, working backward through the chain. He gets to the last device and makes the final change, perspiration beading on his forehead, his gaze fixed on those ping results.

At that very moment, the person who drew the short straw to make the four-hour drive is sitting on her couch and watching TV. She's constantly distracted by her silent cellphone, wondering if it'll ring and send her on a midnight cruise or if she'll be able to finish tonight's episode of "Game of Thrones."

A few pregnant seconds tick by as the timeouts continue, then suddenly bytes are received. The bits, they can pass once again. The admin suddenly realizes he's been holding his breath; now he lets it out in a whoosh, gives himself a mental high-five, and runs a few more tests to make sure everything is normal. Then he sends a very brief email to the IT group stating that the operation was a success and that adding redundant network access to this site might be a good idea going forward. He grabs a beer. The cellphone will remain silent, at least for tonight.

As much as we'd like to think that single-homed networks are becoming rarer with the advent of cheap business-class cable, fiber, and DSL circuits, they're still a reality for many companies the world over. It's simply not always possible to maintain redundant egress circuits to every site a company may operate. The general rule is that the more remote the site, the fewer the options available. 

Back in the days when T1s ruled the roost, remote cutovers were very common, and network admins who cut their teeth during those times know both the thrill of victory and the agony of defeat. These high-wire acts tend to make one very circumspect, and very exacting in one's understanding of the network arts, as failure typically carries a considerable penalty in terms of both network downtime and reputation. 

But without risk, there's no reward. To a network admin, seeing those packets finally come back after those interminable few seconds, that reward is great indeed.

This story, "Mission impossible: A remote network cutover," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

Copyright © 2012 IDG Communications, Inc.