Surgient delivers virtual test labs

Test-lab automation software that leverages virtualization to ease fundamental IT operations has started to emerge.

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft TEST CENTER DAILY BLOG Track the latest product reviews and news from the InfoWorld Test Center. Surgient Virtual QA/Test Management System 5.0 Surgient, surgient.com Very Good  8.0 criteria score weight Capability 9 45% Ease-of-use 5 15% Interoperability 7 15% Scalability 9 15% Value 8 10% Cost: $4,000 per host CPU socket; discounts on large installations Platforms: Windows 2003 Server Bottom Line: VQMS is a capable, enterprise-scale lab-automation solution that exploits virtualization. It boasts a good management console, a scheduling system, excellent capture and replay capabilities, and a helpful reporting system. The extensive feature set, however, is hamstrung by pervasive complexity. About our Reviews and Scoring Methodology

Just two products currently compete in this market space: Surgient VQMS (Virtual QA/Test Management System) and Akimbi Slingshot. I reviewed Akimbi Slingshot software earlier this year and found it to be a good solution. In the intervening period, VMware bought Akimbi and is enhancing Slingshot, which it plans to integrate with its VMware Lab Manager, currently in beta.

VQMS is a powerful, high-end solution that will please IT managers. At its core, it mirrors the capabilities of Akimbi Slingshot, setting up, deploying, copying, and tearing down complex multiserver configurations of virtual machines. Surgient then adds several enterprise-friendly features, such as an advanced scheduling system and a reporting module that’s mostly oriented toward tracking resource utilization. The trade-off to these benefits is that running VQMS is a complex and, at times, frustrating experience.

Surgient’s core business is hosting virtualized apps, especially for training and demos. This hosted model has deeply influenced the design of VQMS.

VQMS has two principal modules. The main console, called VCS (Virtualization Control Server), sits on one server and uses software agents to monitor VMs running on other systems, called hosts. These hosts -- which can run VMware ESX Server, GSX Server, or Microsoft Virtual Server 2005 -- are generally aggregated into pools. Support for Xen is under consideration.

Separate library servers store images of VMs that can be cloned and instantiated on the host systems. These libraries also hold snapshot images taken while a VM is running. Likewise, configurations consisting of multiple VMs can be saved to disk for replay at a later point. The library server can reside on one of the other servers or be standalone -- it’s mostly a storage resource.

The VCS management console is elegant and uncluttered. An admin can easily set up and deploy configurations, as well as manage VMs and related resources after some initial training. Other screens are equally uncluttered, but less intuitive: They require practice to locate important items. And regular usage is needed to recall how actions should be sequenced -- a recurring, but crucial, complexity. An upcoming release is designed to simplify this aspect.

Setting up a configuration involves copying VM images from the library, provisioning them with unique IP addresses, and optionally placing them inside what Surgient calls a NAIL (network abstraction and isolation layer). This feature installs a software router in the configuration that will perform MAC (media access control) and IP address translation. The address translation is important for enabling the saved configuration to run even while the original configuration is still running. Because these configurations cannot have the same MAC addresses or IP addresses, they’re cloaked to the outside world via the software router. Meanwhile, within the configuration, the original IP addresses and MACs are preserved -- a critical feature if an application is tied to a specific MAC address.

At this point, the configuration is still not ready for deployment. A reservation must be made on one of the virtualization servers to deploy the machine. After a reservation has been made, the administrator can run the configuration or release it to a team member for deployment. This reservation system is a key distinguishing feature of VQMS; Akimbi Slingshot has no equivalent. It enables sites to plan usage on their virtual servers. You cannot schedule more jobs than there is available capacity.

The price for this benefit is high, in my opinion. Nearly every operation in VQMS is intimately tied to this concept of reserving resources. For example, if you want to increase the amount of RAM on one of your VMs (something that normally requires a simple restart of the VM), you cannot do so in Surgient without shutting down the entire configuration, reconfiguring it, and redeploying it -- provided a host server allows you this extra RAM.

In contexts such as Surgient’s own hosting business or the training application discussed earlier, this approach makes sense, but in a QA-style lab, I don’t see it. Lab setups frequently require on-the-fly configurations and deployments (for example, QA discovers a bug, captures the VMs at the point of the bug, and sends a link to the developer, who can then run the configuration to see the bug). Likewise, tech-support usage is frequently configured with the customer on the phone. These activities tend to be ad hoc, one-off uses that are encumbered by reservations.

Click for larger view.

And because the reservation system is unforgiving, it is more of a pain in lab contexts than a benefit. Moreover, given the low price of hardware these days, most sites will choose to add a few boxes to their virtualization platforms if they don’t have enough capacity, rather than impose a reservation system. However, true command-and-control IT sites will find in it something to love.

An item that all managers will appreciate, however, is the reporting system, which is elegant and easy to use. It, too, is unique. Akimbi does not have this feature. Picking up on the familiar theme, these reports mostly focus on resource allocation and availability.

Surgient staff normally installs VQMS. This is the right approach. Although the manuals provide instructions on installation, the process is fraught with difficulties. Surgient also recommends training VQMS administrators. This, too, I would recommend. The operations manuals are decent, but it’s simple to become entangled in the details of configuration and deployment. Unfortunately, the manuals have no explanations of error messages nor is there an accessible knowledge base on the Web site, so when a problem occurs you have no option but to call tech support.

Despite these difficulties, after VQMS is installed and you’ve learned your way around, you’ll find it provides a capable, enterprise-level lab-automation solution. It is easy to recommend that it should be evaluated by any prospective customer, because it is one of only two available products. However, even if this were not the case, its comprehensive feature set and enterprise orientation would clearly warrant thoughtful consideration.