Back in the old days, the only realistic way to connect multiple remote sites was by T1 or T3 delivered either point-to-point or via Frame Relay. These were either slow and expensive or fast and unbelievably expensive. Then came MPLS, which dispensed with the need for point-to-point circuits from site to site, but was still bound by high expense. You got what you paid for. These circuits were not only reliable, but if a T1 or T3 circuit dropped, you could generally count on the carrier to jump on the problem quickly and resolve it with some expediency.
As cable and DSL networks began expanding, the ISPs introduced the concept of a business-class circuit. With significantly higher bandwidth than a T1 for far less money, these circuits are quite attractive -- but susceptible to the vagaries of their physical plant, which is to say that they're not as reliable as the T1s and T3s of old. In many cases, that trade-off is acceptable since the cost savings can be measured in the tens of thousands of dollars per year.
However, alternative ways to connect remote sites may enter the picture depending on their physical locations. The ideal solution is to be fortunate enough to find that all of your sites are served by a single fiber carrier, such as Optimum Lightpath. In other cases, you may find that your last-mile carrier (such as Verizon) has fiber to your locations, and a carrier such as Cogent can tie up all those ends into a connection served by a single end-to-end network. This way, you get bidirectional speeds up to 1Gbps between sites served by the same carrier, all for a lower monthly cost than a few 1.5Mbps T1s. If possible, this offers the ability to treat remote sites as local, enabling all kinds of replication and application delivery options that simply aren't available with lower-bandwidth circuits.
The new WAN: Weighing the options
For this case study (see "How to slash your WAN costs" for the background), we're detailing a migration from an MPLS network fed with multiple bundled T1 circuits at each site to 100Mbps fiber circuits installed at two sites with a common carrier and a business-class cable circuit feeding a smaller office. These sites are spread across three states, hundreds of miles apart. The previous design was simple: four T1 circuits at HQ, three at the large remote site, and a single T1 at the small office. Each location was linked to a single MPLS provider to handle internal routing.
The costs of exploratory digging, permits, and other nonsense that would be required to find the other conduits was far too high, so the plans for adding the small office to the fiber network were scrapped. Instead, a business-class asynchronous cable circuit was ordered, and fingers were crossed. Fiber is a dedicated medium, whereas cable is shared. Fiber is also physically more robust than cable, and it generally elicits faster response times when trouble occurs. But cable would have to do.
Once the 100Mbps fiber circuits were built out at the two larger locations, Cisco ASA5510s were procured, configured, and tested. Each site would have an AES-256 VPN tunnel to the other sites, with VoIP and video traffic prioritized to ensure that phone calls and videoconferences would trump all other traffic. Further QoS was implemented to ensure that internal WAN traffic would supersede Internet traffic.
The new WAN: Upsides, downsides
The initial testing showed the best possible scenario: The latency between the two main sites was right around 10ms, roughly a third of the latency on the dedicated MPLS network -- not bad for a 200-mile round trip. The smaller site had somewhat higher latency due to the fact that it was served via another carrier, but was still around 35ms.
The new network was stress-tested and cut over during a weekend maintenance window. The speed bump from 4.5Mbps to 100Mbps for internal traffic was lost on the users, but immediately noticeable to IT, which quickly put the abundance of bandwidth into use for SAN replication and backup consolidation. The voice and video traffic not only ran smoothly, but in fact the videoconferencing resolution could be upped to 1080p without a hiccup. And the monthly WAN costs dropped by one-third, saving nearly $40,000 per year -- talk about a win for IT.