In our continuing effort to create more realistic enterprise simulation, we agreed to use Spirent’s new TestCenter 5000A and combined it with its Avalanche, ThreatEx, and Abacus gear in order to fully load our competitors with the triple play of network traffic: voice, video, and data.
Using the Spirent equipment, we designed real-life traffic patterns based on high-volume throughput from the University of Hawaii production network. We wanted to know if this new generation of 10-Gig hardware had really matured enough to be viable in a standard enterprise environment.
When our participants were done with these baseline performance tests, we enabled an optional testing period. One challenge looked to bang on self-defending network technologies. Vendors could choose to participate in this test or design other tests in conjunction with our lab staff to show off features about their product suites that they deemed important or unique.
For our baseline tests, we had to find out just how much data the intercore links could handle. For this we created Baseline 0, where we hooked up 20 copper gigabit feeds into each of the edge distribution switches and ramped up the best-effort layer 3 data until we started dropping packets. Because any routed network requires some interswitch communication, we needed to quantify just how much bandwidth we were giving up.
Next, we ran Baseline 1. This test sought to confirm that high-priority VoIP and video traffic were actually being protected in the proper QoS queues. For this test, we fired up the Spirent Abacus ICQ boards to create hundreds of VoIP calls coming in from each edge and crossing the core. We then used the Abacus’ PESQ (Perceptual Evaluation of Speech Quality) and 3Tone (three test tones used for stress testing) to measure how well the switches preserved the quality of voice calls.
Next, we used the Abacus VQM (Video Quality Measurement) to monitor the multicast video streaming edge to edge, again using its own metrics to maintain a tight view of video quality. Lastly, we used the Spirent Avalanche to send synthetic application layer traffic through the network targeted at the Reflector unit on the other side of the network. Together, Avalanche and Reflector simulate real application-style network traffic.
Our Baseline 2 combined Baseline 0 and Baseline 1 to guarantee that we were oversubscribing the intercore links while we sat back and watched the smoke. The MOS and PESQ for VoIP quality and the V-Factor for video quality are all measured on a scale of 1 to 5, with 4.5 being a realistic maximum or MOS and for PESQ and 4.9 being the mark for DVD-quality video. R-Factor measures voice quality on a 100 point scale, with 90 to 100 corresponding to MOS scores of 4.3 to 5. The goal was to measure a slice of a midsize corporation’s general traffic flow. We wanted to confirm that each vendor really could protect the voice and video flows according to the standards provisions. As a side benefit, this also gave us a historical view because we’ve actually done variations on this test for the past 10 years.
What we found confirmed our theory that our baselines were reasonable, and for this particular scenario, nearly identical. In our voice tests, both vendors’ gear scored a 4.5 for average PESQ and 93.2 for R-Factor. HP’s V-Factor average was around 4.95; Extreme’s was about 4.91.
The reality is that we had expected no appreciable variances between our test vendors during the baseline performance tests. This is because 10-Gig networking really has matured enough that large variances between switch products in the day-to-day performance department simply don’t happen anymore. Even if Cisco had participated, we’re confident that all the solutions would differentiate themselves only in the optional portion of our shootout.