Evaluating security software is not a walk in the park. It's more like a trip to the circus: Vendors promise the world -- 100 percent accuracy! 100 percent protection! -- for a big one-time fee plus an annual subscription. The one thing you can be sure of is that the results won't measure up to the promise.
If it's your job to choose security software, you have my sympathies. To make things a little easier, I offer my seven-point plan for evaluating solutions. The products have changed radically during the two decades I've been using this framework, but it works. That's why I still swear by it.
[ 6 lessons learned about the scariest security threats | It's time to rethink security. Two former CIOs show you how to rethink your security strategy for today's world. Bonus: Available in PDF and e-book versions. | Stay up to date on the latest security developments with InfoWorld's Security Central newsletter. ]
1. Write down your goals
First, write down the tactical goal you want to meet with a new product. This might seem ridiculously easy at first, but it becomes less so as you get more specific. For example, you might say you "want antivirus software," but at the next level of detail, what you really want is an antimalware program that runs on the server and on clients, with real-time, on-demand, and scheduled scanning. Does the antimalware program also need to offer a host-based firewall, include antiphishing, or send alerts to your help desk?
2. Create a feature list
Here's a sample list of options: What clients and servers must the product protect? What types of servers (Windows, Linux, and so on) must the management and production software run on? You might also want to indicate which database (Microsoft SQL Server, Oracle, MySQL, Hadoop) and Web server (Apache, IIS) technologies you'll accept. Do you want the product or service to protect mobile computers, tablets, and smartphones? Must the protected clients already be managed by your company? Must the clients reside on the corporate network, or can they be located across the Internet? If you put a lot of thought into your feature list, it can easily reach several dozen requirements.
Be sure to indicate which features are deal breakers vs. nice-to-haves. Have the selection stakeholders (hopefully including both management and end-users) review and approve the feature set. Get everyone to agree to the deal breakers.
3. Do your research
I'm a big fan of software reviews. Rarely do I read one and fail to come up with an issue or feature I would have missed had I tested the software before reading the review. Often, flaws noted in review articles will be fixed by the time you personally review the software, but knowing what was problematic and how the vendor fixed it can help in your decision.
4. Create a test environment
I highly recommend using an isolated test environment before performing even limited testing in your production environment. The biggest decision to make is how much effort you want to put into creating a test environment that accurately mimics your production environment.
For example, next week I'm doing a demo of a set of products that in the production environment will require seven different servers. Do you want to mimic all seven servers in the test environment, or is it acceptable to merge roles? If you can, you always want to mimic the configuration that will go into the production environment, but you can sometimes get away with fewer machines (or VMs) without sacrificing accuracy.
Think about naming conventions in the test environment. I usually recommend that the test environment names should be at least slightly different than those of the production environment. Why? Because many times the supposedly "isolated" test environment ends up having one or more connections into the production environment; if you use the same names, you could cause real operational issues.
Write down how you configured your test environment with enough detail so that anyone relying on your results could re-create it. If you're using virtual machines, now would be a good time to take a VM snapshot. That way if you're testing multiple products for the same solution, you can make sure you start with a clean slate for each test. I also usually take another snapshot just after the computer security software is installed, so you can go back to the original install state if you decide to change configuration options and test again. Create at least one test image for each platform the product must support.