A strong core VA (vulnerability assessment) engine is important, and we ran each vendor’s engine through its paces on our networks. We tested our two separate networks, one in Sunnyvale, Calif., and the other in Monterey, Calif. under a variety of different conditions and loads.
On top of testing the engine, we focused on a variety of VA capabilities, specifically management, reporting, speed, accuracy, and security. We also took a look at how Qualys and Foundstone fared against their competition, including the open source Nessus Project, Version 2.0.6, and the new Internet Security Systems’ Internet Scanner 7.0.
Our two test bed platforms and operating systems were diverse to say the least, with machines running all flavors of Unix, Linux, and Windows. We had several versions and types of non-Windows operating systems, including NetBSD, FreeBSD, OpenBSD, SunOS, BSDI, and Solaris; Linux, Irix, Mac OS, HPUX, RISCOS, Minix, BeOS and v2OS. Windows versions included a number of Windows 98, 2000, NT, and XP machines, and lastly, a number of devices running Cisco IOS.
We placed a heavy emphasis on management of the vulnerability application and appliance, and kept an eye towards useful results and reports after VA scans. Of course, our data's security was important as was the ability to run a VA scan against a machine without experiencing a crash in the process.
In terms of management, we looked over how the user interface was designed to the task at hand, and the ability to easily troubleshoot and maintain over time. Products with a Web-based management infrastructure scored well, as did products that supported multiple appliances from a centralized management interface. Having the ability to update the VA database in a timely fashion was also extremely important.
We looked at reporting with a critical eye toward the presentation of useful VA statistics. If the product had a useful presentation of data, especially with functional graphs and trending over time, the product scored better. The product had to have a way to verify vulnerabilities and include some information on a patch or steps to fix vulnerabilities. The ability to display network maps, whether in text or graphical format, brought scores higher. We also looked at the ability to create practical e-mail or paper reports.
Securing vulnerability data was also critical. Network and host vulnerability information reaching the public domain would be disastrous, so we looked for multitiered authentication to limit VA data to authorized personnel, security of the data during transport from management machine to the user interface, and the ability to manage VA reports securely.