Test Center guide: Mail security appliances

Mail security solutions differ in anti-spam techniques, accuracy, false positive rates, and ease of setup and administration. We compare Barracuda, BorderWare, Cisco IronPort, Mirapoint, Proofpoint, Secure Computing IronMail, Sendio, Symantec, and Tumbleweed

E-mail security continues to be a hot-button issue for IT administrators, who now find more moving parts in mail security solutions than they did just a couple of short years ago. Fighting viruses and spam were the original spurs for creating e-mail security appliances, and anti-spam is still the most important component of mail security. But the solutions have evolved to meet a host of additional requirements. These include securing connections between users, both internal and external; preventing loss of corporate data; stopping new types of threats such as phishing, spyware, and other types of malware; and blocking DoS and other network attacks as well as some application-layer attacks on mail servers.

There are three basic types of e-mail security solutions on the market: software-only, appliances, and hosted solutions. Software-only products range from free, such as the open source SpamAssassin, to quite expensive, but there are relatively few software-only solutions, primarily because setting up all the necessary software is complex and easy to get wrong. Thus, most vendors provide an appliance to run their software on, greatly simplifying the installation process. Appliance installation is generally a matter of setting basic network information and telling the appliance where to send e-mail once it's been filtered.

Hosted e-mail security solutions work on a different model. The Internet DNS (domain name service) settings that point to your e-mail server are changed to point to the service instead. The service receives all e-mail sent to your domain and forwards the good stuff to you, filtering out spam and viruses. One advantage to a hosted solution is that the volume of mail coming to your internal network is greatly diminished – by 80 to 90 percent in most cases. Also, because the bad e-mail is never received at your location, you need not worry about archiving it, which might be an issue if you're doing the filtering in-house.

All the solutions reviewed in this guide are appliances. Services will be added later. Due to the time necessary to allow DNS changes to propagate and other factors involved in testing, it isn't practical to mix testing of appliances with hosted solutions.

Choices in mail security
Feature checklist
Data protection options
Testing mail security appliances
Mail security appliance reviews
Barracuda Spam Firewall 400 (v3.4.10.102)
BorderWare Security Platform SP-800 (v7.1)
Cisco IronPort C100 (v5.5.1)
Mirapoint RazorGate 160 (v3.8.4-GA)
Proofpoint Messaging Security Gateway P840 (v5.0)
Secure Computing IronMail E2000 (v6.5.2)
Sendio I.C.E. Box (v3.0)
Symantec Mail Security 8340 (v7.5)
Tumbleweed MailGate 5650 (v3.1.2-4366-HF1)

Choices in mail security
Choosing an appliance means more than selecting the highest filtering rate. The easiest way to stop all viruses and spam is to stop all mail; the trick is to stop as much of the bad mail as possible without stopping any of the good mail. This has gotten much harder over the years. Because the spoils belong to spammers who get their message through, spam evolves quickly to bypass new filtering paradigms. As with anti-virus technologies, spam is a moving target, requiring constant updates to filtering rules.

You may also find that you and some vendors disagree on what constitutes spam or malware. A number of the vendors – Barracuda Networks, BorderWare, Mirapoint, Proofpoint, Secure Computing, and Sendio – stopped many marketing e-mails and other types of bulk e-mails that users may have signed up for, leaving it to the individual user to add senders to the whitelist. Because all of the messages that were blocked were messages I'd signed up for – product updates, newsletters, weekly specials from vendors I use, and so on – they were all counted as false positives. However, I also whitelisted each bulk e-mail when it was stopped, so the total bulk false positive represents the number of unique senders that were stopped; no duplicate bulk e-mails were counted as spam.

Lots of bulk e-mail doesn't comply with the CAN-SPAM Act, which requires that the "from" address and sending domain match, among other things – so that mail from xxx@infoworld.com comes from a server in the xxx.infoworld.com domain. Many organizations outsource their bulk e-mailing to third parties, who don't bother to set up the domains correctly. For example, a bulk e-mail (newsletter) from Secure Computing Magazine has a sender address that isn't SCmagazine.com, or even haymarketmedia.com, but bull_05_sc_01112006@ecm.hbpl.co.uk. In other cases, e-mail newsletters from legitimate senders such as infoworld.com come from a different address each time. Thus, you need to whitelist the domain, rather than the sender, which creates the potential for spam that is apparently from that site to make it through.

Some administrators may attach minimal importance to whether or not users can receive bulk e-mail, but some of these messages include security updates from vendors such as Red Hat and Microsoft. Personally, since other products match the catch rate while blocking far fewer legitimate bulk messages, I think the problem is solvable in other ways. A couple of products offer two levels of filtering: They classify messages as spam, bulk mail, or legitimate, rather than either spam or legitimate, allowing users to sort bulk e-mails into a folder for occasional perusal.

In terms of installing a system that will have a minimal impact on end-users, the rate of false positives is more important than the catch rate for spam. If users find they aren't receiving messages they're expecting, they'll spend as much or more time looking through the quarantine than they would deleting spam in the first place.

Similarly, some anti-malware products may stop programs that exhibit behaviors similar to adware, even if the user wants the service that comes with the program. In these cases, management will have to make the call as to whether users should be able to whitelist these programs themselves or whether they will have to go though the administrator. The latter gives the admin better control, but may leave them handling dozens or hundreds of requests, depending on the number of users and how stringent the filtering rules are.

One differentiator among appliances is the ease of configuration and maturity of the interface. LDAP configuration is particularly problematic. All the devices tested could import information from Active Directory or other enterprise directory servers to verify that incoming mail is addressed to valid recipients. However, depending on the product, LDAP setup could be a matter of a few clicks, or a long and involved process of trial and error to get the syntax of the LDAP queries correct.

Choices in mail security
Feature checklist
Data protection options
Testing mail security appliances
Mail security appliance reviews
Barracuda Spam Firewall 400 (v3.4.10.102)
BorderWare Security Platform SP-800 (v7.1)
Cisco IronPort C100 (v5.5.1)
Mirapoint RazorGate 160 (v3.8.4-GA)
Proofpoint Messaging Security Gateway P840 (v5.0)
Secure Computing IronMail E2000 (v6.5.2)
Sendio I.C.E. Box (v3.0)
Symantec Mail Security 8340 (v7.5)
Tumbleweed MailGate 5650 (v3.1.2-4366-HF1)

Feature checklist
Nearly all AV engines use a combination of signatures that are constantly updated by the vendor, along with heuristics that attempt to identify dangerous attachments that aren't caught by the signatures database. Anti-spam techniques include sender reputation, based on the vendor's database of IP addresses known to be sending spam; certain TCP/IP tricks such as requesting a resend of the message (legitimate mail servers will resend, while most spam engines don't); heuristics of many different varieties; and a host of other specialized techniques, including such oddities as employing optical character recognition to identify image-based spam that doesn't use conventional text in the message. Filtering and spamming techniques evolve through a constant battle between the anti-spam vendors and spammers, who are desperately trying to slip their ads past the filters. Because the spammers are commercially motivated to bypass new heuristic techniques quickly, many vendors are relying more on reputation-based filtering.

While anti-virus and anti-spam are the essence of mail security, there are a number of other features you should expect to find in all e-mail security appliances. These include:

·         Policies that can be set per user, per group, or per site to control when users can send and receive mail, to whom, whether whitelists or blacklists can be modified by users or admins, which types of attachments are allowed on incoming and outgoing mail, and so on.

·         Support for multiple domains or back-end mail servers.

·         "Outbreak" anti-virus, which is designed to snare viruses for which signatures don't yet exist. Outbreak AV filters typically stop messages that have the characteristics of a virus, such as an executable attachment or a suspicious origin, then review them over the next 24 or 48 hours to see if a signature appears; if not, they notify the user or admin to inspect the message and release or delete it.

·         Secure content management features that examine outbound messages for specific phrases, types of files, or specific file names, and log or quarantine them for review.

·         LDAP/Active Directory synchronization.

·         DoS protection, which blocks repeated attempts to ping, send connection request, send directory request, send user verification, or basically any type of request for a response from the server that exceeds a certain frequency threshold, such as more than 100 pings per minute from a particular IP address.

·         Directory harvest protection, which is designed to thwart attempts to send messages to all possible addresses on a mail server. By discovering which addresses are not rejected, so-called directory harvest attacks attempt to build a database of valid addresses. To combat this, when the appliance sees a large number of messages going to invalid addresses, it either throttles the connection (limiting the sender to one message per minute, for example) or blocks that IP address entirely.

·         Address verification, to block e-mails sent to nonexistent users, and the ability to use reverse DNS to verify that a sender's IP address matches the sender domain. The use of reverse DNS thwarts phishing attacks by preventing forged e-mail from getting through.

Choices in mail security
Feature checklist
Data protection options
Testing mail security appliances
Mail security appliance reviews
Barracuda Spam Firewall 400 (v3.4.10.102)
BorderWare Security Platform SP-800 (v7.1)
Cisco IronPort C100 (v5.5.1)
Mirapoint RazorGate 160 (v3.8.4-GA)
Proofpoint Messaging Security Gateway P840 (v5.0)
Secure Computing IronMail E2000 (v6.5.2)
Sendio I.C.E. Box (v3.0)
Symantec Mail Security 8340 (v7.5)
Tumbleweed MailGate 5650 (v3.1.2-4366-HF1)

Data protection options
For organizations seeking additional message protection, there are two kinds of encryption available, which can help secure e-mail sent between corporate sites or between you and your partners: TLS (Transport Layer Security) and per-message encryption. TLS is encryption from server to server between domains. It requires setting up an encrypted connection in advance, and then any e-mail between those two servers will be encrypted. Per-message encryption uses PGP or some other algorithm to encrypt individual messages. The user at the other end must have the proper key to decrypt the message. This doesn't require advance setup, but users receiving encrypted messages may not be able to decrypt the message without help from an admin.

Most appliances can provide TLS, and a few can also provide policy-based encryption using a separate encryption engine. One encryption engine, the Voltage IBE (Identity-Based Encryption), which is available with the Proofpoint and Secure Computing IronMail appliances, makes decryption very easy for the recipient; the user need only click a URL in the message and enter their information, and they are then able to decrypt the message. (See our review of the Voltage SecureMail Appliance.)

If you are interested in checking incoming or outgoing messages against specific word lists to spot potential data breaches or ensure compliance with HR requirements, you should be aware that some vendors make this easier than others, by providing a GUI rather than requiring you to refer to a manual to find the exact syntax to type in. Further, not all vendors provide standard lists of words along with their content management engines. But nearly all the vendors (Sendio is a rare exception) will monitor incoming and outgoing messages, and block on specific words or phrases or patterns. And if they don’t provide lists themselves, they will allow you to create or import lists of words in a number of categories, such as sexually explicit language, otherwise offensive language, politically incorrect language, terms that might refer to proprietary intellectual property, and phrases or numbers that could violate confidentiality agreements or legal requirements, such as a customer's Social Security number or credit card information. If a product you otherwise like doesn't include such lists, you can often find them on the Internet.

Appliances come with a wide variety of backup capabilities, ranging from Mirapoint's compatibility with commercial backup software such as NetBackup, and Tumbleweed's automated backup of the mail store, to the minimal ability to save a configuration file or perform a complete backup of the mail store by typing a Linux tar command at the command line. If you will be maintaining a message quarantine on the appliance, you may want a solution that allows you to back up the quarantine separately.

Also worth asking is whether the secure content manager can scan inside zip files, or detect encrypted attachments and block or delete them. All the appliances tested except the Sendio could block messages that contain zip files (or executables, or any other attachments, either by extension or file size). And all except the Sendio, which doesn't do filtering of any kind, could block encrypted attachments. The features table provides an at-a-glance comparison of what each product provides.

Choices in mail security
Feature checklist
Data protection options
Testing mail security appliances
Mail security appliance reviews
Barracuda Spam Firewall 400 (v3.4.10.102)
BorderWare Security Platform SP-800 (v7.1)
Cisco IronPort C100 (v5.5.1)
Mirapoint RazorGate 160 (v3.8.4-GA)
Proofpoint Messaging Security Gateway P840 (v5.0)
Secure Computing IronMail E2000 (v6.5.2)
Sendio I.C.E. Box (v3.0)
Symantec Mail Security 8340 (v7.5)
Tumbleweed MailGate 5650 (v3.1.2-4366-HF1)

Testing mail security appliances
I tested appliances from nine top vendors: Barracuda Networks, BorderWareTechnologies, Cisco Systems, Mirapoint, Proofpoint, Secure Computing, Sendio, Symantec, and Tumbleweed Communications. (See the links to each review below.) I tested each product with a real, live e-mail stream over 15 days, exposing each to 13,000 to 14,000 total messages, consisting of about 2,500 real messages and 10,000 spam messages. Because spam evolves very rapidly, and anti-spam signatures strive to keep pace (often removing old definitions to keep the database as small as possible), collecting a corpus of spam over a period of several months and then replaying it isn't a valid test of filters. Further, because most anti-spam vendors use the sender's IP address as a critical part of their detection of spam, replaying a collection of messages and spam from a single IP address renders one of the primary detection mechanisms useless, causing much lower scores for engines that would perform well in the real world.

The results chart shows that some appliances received a smaller number of spam messages, from 1,969 at the lowest, between 5,000 to 6,000 at the middle of the pack, and more than 10,000 for two products. This disparity in numbers of spam received is due to the fact that all of the appliances reject varying amounts of spam without accepting and filtering it, based on the sender's IP address and other factors. The average number of spam messages sent to the mail server is about 13,000 to 14,000 per two-week evaluation period. The number of messages caught by pre-filtering varied from 3,000 to 4,000 for the Proofpoint and Tumbleweed products, to 10,000 for the Barracuda.

Comparing the filtering rates is not terribly important. Only two solutions scored less than 95 percent: the Cisco IronPort and Barracuda Spam Firewall appliances. The Cisco, at 93.4 percent, and the Barracuda, at 88.4 percent, still fall well within useful catch rates. More important in terms of impact on users is the percentage of false positives, which is excellent in the case of Cisco IronPort, and not so good for the Barracuda.

Because e-mail retention policies may require that any mail received be archived, appliances that reject spam without receiving it – by refusing the sender's invitation to communicate – can dramatically reduce the amount of traffic on the internal network and the load on the appliance itself. It also reduces the amount of mail that must be archived for e-discovery or other requirements.

In some instances, the messages that are rejected are logged, in which case you might want to follow the logs for a couple of weeks to ensure that no real messages are being rejected. With other products, there's no way to know what's being rejected; you simply have to trust that the pre-filtering mechanism is not rejecting messages from legitimate senders.

In addition to testing anti-spam performance, I tested each product with a stream of current viruses provided by two anti-virus vendors, then tested all mail that wasn't stopped with four different anti-virus clients. The good news here is that none of the appliances allowed any viruses through, or at least none that were detected by any of the four anti-virus engines.

In addition, I looked at anti-phishing and anti-malware performance. The news here is not so good; the anti-phishing filters stopped between 51 and 82 percent of phishing messages, and often blocked legitimate messages from potential phishing targets. For example, some filters failed to block bogus messages that purported to come from www.citibank.com, and blocked legitimate messages from another bank.

Finally, I looked at secure content management capabilities. This is difficult to measure quantitatively, because filtering on keywords tends to either work or not work. However, there are some important differences among the products, principally in the number of different types of files that can be scanned, especially zip archives and other compressed files and archives, and their handling of encrypted files. Some products can detect encrypted files and either hold them for inspection by an administrator before allowing them through or at least keep a copy for later inspection. 

The product that is the best fit for you will depend on your specific requirements and what you are willing to expose your users and your customers to. My pick for the best performer was not the product that filtered the highest percentage of spam (the Sendio I.C.E. Box), nor the product with the fewest total false positives (the Cisco IronPort), but the one with the best combination of accuracy and other characteristics (Symantec Mail Security). In addition to offering great filtering accuracy, the Symantec product is very easy to configure and administer, and sports one of lowest prices (price of the appliance plus the cost per user for 1,000 users).

Mail security appliance reviews
Barracuda Spam Firewall 400 (v3.4.10.102)
BorderWare Security Platform SP-800 (v7.1)
Cisco IronPort C100 (v5.5.1)
Mirapoint RazorGate 160 (v3.8.4-GA)
Proofpoint Messaging Security Gateway P840 (v5.0)
Secure Computing IronMail E2000 (v6.5.2)
Sendio I.C.E. Box (v3.0)
Symantec Mail Security 8340 (v7.5)
Tumbleweed MailGate 5650 (v3.1.2-4366-HF1)

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies