Pauper's guide to systems testing

You don't have a hardware lab for network and system deployment testing? You may not need it

If you have access to a system and network infrastructure testing lab, fully stocked with server and network hardware, consider yourself incredibly fortunate. The predominate corporate attitude has always been that infrastructure testing isn’t nearly as important as development testing, and since it tends to be expensive, it’s not going to happen. This leaves network and systems folks with much more seat-of-the-pants work than anyone else in IT or development -- that’s just the way it is.

Most of us acknowledge we’re unlikely to put together a full lab with enterprise-grade hardware and software that mimics production to test rollouts and infrastructure changes, but we also know we can’t improvise these operations either (most of the time). We come up with ways to test things that aren’t as close to production as we might like, but go a long way to ferret out issues with a planned design or deployment.

For instance, if the goal is to configure and test edge security devices for a rollout to a pile of remote sites and you’re a network guy, you should know enough about Unix to build a VM or laptop into an Internet mimic and stage all the hardware in one place until everything is tweaked and tested. If you don’t have the budget for testing gear from a vendor such as Ixia, you should be able to put together a mockup to at least saturate network links and test interconnections, fail-over, and so forth. Open source tools such as packETH and iPerf, or even good old Netcat, can take you pretty far in this type of testing.

As an example of a simple testing method, if we needed to stage and test a half-dozen remote site firewalls that will be deployed, we could toss up a Linux VM (or a laptop) connected to a test network on one interface, and to a routable network with Internet access on another interface. A switch connected to the test network would be populated with the WAN links from each of the firewalls. The test VLAN interface on the Linux VM would then be configured with the defroutes of each staged firewall, IP forwarding enabled, and an iptables NAT configuration introduced on the routed interface.

This takes very little time to set up, and suddenly you have simulated the WAN, MPLS network, or Internet for all of those endpoints. If you set up some dummy hosts on the internal interfaces of those firewalls, you can simulate LAN hosts. You can now configure and test a meshed VPN configuration, fail-over, internal LAN functions, or anything else, and go to production fairly carefree.

You may also want to test by simulating lower-bandwidth circuits or congested spotty network links. This can also be done within that Linux VM. For starters, most people don’t know you can use iptables to simulate lossy connections with randomized dropped packets. You can introduce this with the following command: 

iptables -A INPUT -m statistic --mode random --probability 0.01 -j DROP 

This will drop packets with a 1 percent probability. Naturally, you could also do that while introducing fixed or randomized latency using netem. The command to introduce randomized packet loss with netem is simple:

tc qdisc change dev eth0 root netem loss 0.1%

You can get much funkier from there, with the ability to introduce packet duplication, delays, reordering, and more. It’s a wonderful and simple tool that can be a huge help in testing planned deployments or even trying to debug extant problems.

If you need to restrict bandwidth in one of these scenarios, there are myriad available tools to do this. Ethtool is the most obvious, perhaps, but that might not be granular enough. Luckily, there are plenty of others.

Wondershaper is quite old, but is still kicking and available as a package for most distros. I’ve found trickle to come in handy from time to time, as it provides a simple way to set fairly complicated traffic control rules from userspace. And of course there’s CBQ, which is a tried and true way to construct complex traffic management rules. Depending on how much time you want to spend or if this kind of thing will actually wind up in production somewhere, you may also want to check out how cgroups can be used to manage bandwidth -- but for a quickly retooled test bed, that might be too deep.

The upshot is we’d all love a big lab with all the toys and all the shiny commercial testing tools. Some of us have those facilities right at hand -- but some of us don’t. That doesn’t mean we forgo testing. It means that we need to get a bit creative with our tools and ideally learn a few things along the way ... which is half the fun of testing, right?


Copyright © 2015 IDG Communications, Inc.