5. Structure and perform rigorous tests
Once you get a test environment locked down and the product in hand, it's time to begin the evaluation. You can get some pointers by reviewing past InfoWorld Test Center reviews. Our reviews tend to break up product evaluation into a handful of categories:
Installation. You'll be evaluating installation on both the client and server side. How many different install methods are supported? Can you push out installs within the product itself or do you have to accomplish that using another method? What services must be running and what firewall ports must be open in order to install the software remotely? If you test installs, how many failed? Be sure to install at least once for each desired platform and form factor.
Configuration. Note how hard the product was to configure during or after the install. The best products walk you through a series of wizards that help you make the best choices. How hard is it to change settings afterward? How long before the changes take effect? Does the product offer both agent and agentless installations?
Management. How do you connect to the management console? Hopefully it's over a HTTPS or other secured connection. Does the management console allow you to create different access control views for different types of administrators? What remote clients does it support?
Logging and alerting. Every computer security product should do lots of logging. What format are the logs in? Can you manipulate the log format, data collected, and export to other formats? Can the logs be saved to an external database, emailed to people, and restarted or recycled on a scheduled basis? How many different ways can alerts be sent (email, SMS, pager, network message)? Does it "message throttle" alerts so that dozens or more aren't sent for a single related event? Does the product allow you create custom alerts or to ignore alerts you don't care about?
Reporting. Most products shine or show their weakness in their reporting. You want a product with lots of built-in, canned reports. The product should allow customization of current reports and easy creation of new ones. Can reports be extracted; if so, to what formats? Can reports be scheduled on a periodic basis and automatically emailed to interested stakeholders?
Performance. Performance can be the hardest measurement to take in a test environment, since you won't have the same scale as the production environment. But don't simply rely on the vendor's claims. Interview other vendor clients of similar size to hear what they have to say about the product and its performance.
Make sure you get in writing what hardware specifications will get you peak performance. Many buyers put a clause in their vendor contract that requires a certain minimum guaranteed level of performance, both on server and client side, or they get their money back.
Support. Most vendors claim to have 24/7 support. But does calling the support number result in having to navigate an endless phone tree? Are the support engineers well trained? How many support calls are you allowed each year? What does it take to get escalated to advanced tiers of support? What would it take to get the vendor back onsite to resolve an operational issue? Again, don't rely on vendor claims. Talk to other customers about the same size as your company. Bigger customers often spend more money with the vendor, which often results in "special" care that a smaller customer may not receive.
6. Get references -- and call them
Ask the vendor for a list of customer references, hopefully of the same size and in the same industry as your company. Although customers are nearly always selected for their undying love of the vendor, I find you can coax actual experiences from them.
My favorite trick is to wait until the end of the conversation, after they've told you nothing but great things about the product. Then close by asking, "What do you wish the product did or did better?" Often, customers will proceed to share their real concerns. I've learned about many deal-breaking problems this way.
7. Test in production
Finally, test your prospective buy in your production environment before committing to a purchase. Nothing surfaces more bugs than moving from test to production.
My last testing hint: Don't be awed by a product just because it's an appliance. An appliance is nothing but harder-to-update software.
Go forth and good luck!
This story, "7 steps to choosing security software," was originally published at InfoWorld.com. Keep up on the latest developments in network security and read more of Roger Grimes's Security Adviser blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.