X-ray vision into your virtual networks

Network Instruments' virtual taps bring visibility to the otherwise hidden traffic traversing the VMware virtual switch

What's not to like about virtualization? It's green, it saves rack space, it even eases remote access issues. Well, one thing virtualization doesn't do is make troubleshooting easy! For example: If you have an application server, a middleware server, and a SQL server all talking over the virtual network, how in the world can you get a glimpse into their conversations? It's not like you can slap a tap on the gig port and fire up a protocol analyzer, because the virtual network never pokes its head into the physical world.

I and other members of the Interop NOC team ran into this exact issue when Neal Allen of Fluke Networks started looking at logical places to install network troubleshooting gear during Hot Stage for Interop Las Vegas 2009. He was carrying a bunch of Net Optics taps and suddenly stopped dead in his tracks, a quizzical look on his face. He then started asking around: Just how do you tap the virtual network under VMware? What happens if we have problems between servers that only talk over the virtual network? If you slap a full analyzer into a virtual machine, have you now changed the environment by sucking up a bunch of resources to observe the virtual network?

[ Cut straight to the key news for technology development and IT management with our once-a-day summary of the top tech news. Subscribe to the InfoWorld Daily newsletter. ]

As I dug deeper into these questions, I remembered a conversation with a friend about the migration of his SaaS application from an internal datacenter to Amazon EC2/S3. He gave me this advice: If your application is well developed and no longer in the troubleshooting stage, clouds can be wonderful. But the cloud isn't a friendly place for troubleshooting. Trying to track down bugs in the growing cloud-based system was costing him oodles of time and money.

So just how do you get a look inside your virtual network? I stumbled across the answer during an update session by Network Instruments on the Observer product line: virtual taps. If you set a VMware virtual switch (vSwitch) and virtual network adapter (vNIC) to promiscuous mode, the vNIC will receive all traffic that flows through the vSwitch. Network Instruments' virtual tap, called vTaps, is software that installs inside a VMware virtual machine, collects all of this vSwitch traffic, and directs it to a physical NIC and into a Network Instruments probe.


Interestingly enough, the promiscuous mode of VMware vSwitches and vNICs has long been the source of misinformation and controversy. It even bit the SE in the Network Instruments seminar, who commmitted the same mistake that many others have made. It would seem that the vSwitch under VMware can be set in an all-or-nothing promiscuous mode, but the more secure solution is to set up port groups and only set specific groups to promiscuous mode. Otherwise, promiscuous vNICs will be privy to network conversations they have no business hearing. The misconception that you can only set the entire vSwitch into promiscuous mode is the cause of a lot of confusion around VMware security.

Note that vTaps are not a separate product, but rather an extra doodad in the Network Instruments probe product line. Each probe and its vTaps require a separate license, but the license allows you to place vTaps in as many places as you need visibility, regardless of whether the traffic is on a virtual network or physical network. Most of all, vTaps are a lightweight way to see inside a virtual environment without impacting its performance and behavior. Let your external tools aggregate the data off the vTap and combine them with the data coming from physical taps.

Naturally, a virtual tap works only if the virtual switch can be put into promiscuous mode. VMware can be set up for either all ports or just certain port groups. Alas, Windows Server 2008 Hyper-V doesn't have this capability yet, though some Hyper-V users clearly want it.

The Network Instruments folks have a long history of leading the pack in capturing and analyzing network-level data in amazingly fine detail. My favorite tool is where Observer displays both halves of a TCP conversation with timing. The Network Instruments Virtual Tap (aka vTap) is yet another sharp blade in the Network Instruments Swiss Army Knife that is the Observer platform.

Test Center virtualization reviews:

HP BladeSystem Matrix

VMware vSphere 4
VMware View 1.0

Citrix XenDesktop 2.0
Microsoft Hyper-V and System Center Virtual Machine Manager
VMware Infrastructure 3 with VMware ESX 3.5 and VirtualCenter 2.5