From spam-busting to content management, we test the limits of hosted e-mail security offerings from AppRiver, MessageLabs, Microsoft, MX Logic, SECNAP, and Trend Micro
E-mail is the primary conduit of information for many organizations, both internally and with the outside world. Unfortunately, e-mail is also a prime channel for annoyances such as spam, as well as security threats in the form of viruses, spyware, phishing attacks, and more. Some companies choose to defend their e-mail systems in-house by deploying e-mail security products. Others, however, look to outside assistance in the form of hosted e-mail security services.
As a follow-up to my review of nine e-mail security appliances, I looked at six hosted e-mail security solutions from AppRiver, MessageLabs (now owned by Symantec), Microsoft, MX Logic, SECNAP Network Security, and Trend Micro. Postini (now owned by Google), another major player in the hosted e-mail security space, declined to participate.
[ Compare these e-mail security services by features. Compare their spam filtering results. See the Test Center guide to e-mail security appliances including reviews of offerings from Barracuda, BorderWare, Cisco, Mirapoint, Proofpoint, Secure Computing, Sendio, Symantec, and Tumbleweed. ]
Before delving into the review itself, however, it's useful to understand just why an organization might want to hand over e-mail-security duties to a third-party provider. After all, many IT managers may be uncomfortable with having a critical application such as e-mail outside their control. However, hosted e-mail services hold several advantages over their in-house counterparts. There are drawbacks as well.
Appliance versus services
One advantage that a service holds over an in-house solution is that when you're using an outside provider, the volume of mail coming to your internal network is greatly diminished -- by 80 to 90 percent in most cases. Moreover, because you don't receive the unwanted mail at your location, you have no need to worry about archiving it. Such might not be the case if you're doing the filtering in-house.
A second advantage of a hosted service is that most providers have more robust networks than even large organizations, with multiple sites that have at least two separate Internet connections and multiple servers. Thus, going with an outside provider greatly decreases the chances that your e-mail service will be unavailable.
Third, hosted services offer a buffer for your e-mail system. If your internal e-mail server fails or if your Internet connection goes down, your mail will continue to accumulate on the hosted service's server until your in-house problem is resolved.
Yet another point for hosted services: They offer all features you'll find in appliances, such as content management, but they also boast services that don't come with an in-house box, including archiving, disaster recovery, and encryption. Bear in mind, however, that if you're going to implement certain features among internal users -- for example, to enforce policies on e-mail content between users in the same department -- you'll have to ensure that all messages are forwarded through the hosted service. This can be complex to set up and could increase delivery times.
E-mail security appliances aren't without their advantages over hosted services, however. Take, for example, directory synchronization. If you want to ensure that e-mail addressed to invalid users is turned away (which you should), you need to export your Active Directory information or user information from another source, be it LDAP, NIS, or something else. Although this is easy to accomplish with appliances, it becomes more difficult with a service. There are two alternatives: Option one is to open a port in your firewall for LDAP (usually port 389). This creates a security hole, however, to which your network admins may object. Option two is to export the data using an application provided by the hosting service. Although I was able to get this process to work during my testing, it took much longer than it did with my previously tested appliances -- up to a couple of hours more.
There's another important drawback to hosted services: When you sign on with one, you'll need to change your DNS records so that mail addressed to your domain goes to the service rather than your internal mail server. The service then forwards the non-spam to the internal server. Any e-mail server that performs a DNS lookup before sending mail to your users should be going with the new address within 72 hours; however, some servers, both spam and legitimate, send messages directly to an IP address and don't resolve the hostname beforehand. These e-mail messages will continue coming directly to your e-mail server, bypassing filtering, unless you configure your firewall to block all incoming e-mail from addresses other than the service. The problem here is that some of the services have multiple IP addresses from which e-mail may be sent, and depending on the firewall, setup can be complicated.
The final drawback with services lies in how user accounts are set up. Users must access the service Web site to view the quarantine, from which they can release messages and (in most cases) whitelist or blacklist senders. Some services can pull account information from Active Directory so that the user logs in with the same password recognized by his or her Windows Domain account. Others offer self-enrollment, forcing users to create an account the first time they log in.
By contrast, appliances generally work with plug-ins to Outlook so that users can review the quarantined messages within their familiar e-mail app or via a local Web site that takes the same log-in and password as their standard Windows account.
E-mail security at your service
I tested the six services in this roundup with a real e-mail stream over 15 days, averaging 16,000 to 19,000 total messages. Of those, about 2,500 were legitimate. The services tracked all incoming messages; thus, I didn't witness the reporting disparity I saw with appliances. How each service counts messages does vary slightly, however. The greatest variable is the number of messages assumed to have been delivered per connection. If a mail server connects to your domain and sends an SMTP message, it may be for one user or multiple users. Most reporting tools assume a message count higher than one, but the actual number assumed varies.
Comparing the filtering rates among the services is not terribly important: As you can see from the results table, they all scored between 94 and 98 percent. (That figure might be more for users with high volumes per day.)
However, the numbers of false positives (legitimate messages mistakenly marked as spam) is far more important to monitor: If users don't trust the system to forward all their important mail, they'll spend more time perusing the quarantine than they would have spent dealing with the full volume of spam. For my testing, most of the false positives were bulk mail, either newsletters or marketing e-mails from legitimate senders who were given permission to send e-mail to the test account. Critical false positives were messages sent from a single legitimate user via a normal e-mail program, which shouldn't ever be identified as spam.
As noted, these services don't just squelch spam. In my testing, they all succeeded in stopping viruses. Anti-phishing was less reliable, with only 60 to 80 percent of phishing messages identified as such, although most were stopped as spam. Features such as archiving, content management, and data recovery all worked as advertised.
When all was said and done, MessageLabs came out with the highest marks in my tests, not only because the service's false-positive performance was best overall, but because it had no critical false positives. Its overall feature set, ease of use, and interface all contributed to the win as well. The second- and third-place vendors, respectively Microsoft and Trend Micro, each had one critical false positive, along with slightly higher overall false positive scores. In the real world, the differences between first and third are nearly indistinguishable. To its credit, Trend Micro is much less expensive ($2.16 per month per user versus $1.60 per month per user), and it offers more features at this price point, along with a guaranteed SLA (service level agreement).
The SecureTide services offer a decent number of features, including spam filtering, virus filtering (drawing on four systems), content filtering, and unlimited queuing. Though easy to set up and use, SecureTide proves lacking in some areas. General spam-stopping performance, for example, is at the bottom of the six services I tested, although still acceptable, with 95 percent of spam blocked. It also suffered 94 false positives and three critical false positives (see test results). The policy engine isn't as robust as some enterprises might like. Also, the service has some irritating quirks, such as requiring admins to whitelist messages one by one.
SecureTide proves simple to set up, as do all the services in this roundup. When setting up users, you have a couple of options: You can import users from Active Directory or another LDAP directory, or you can enter the information manually or through a comma-delimited file.
Once the service is configured, a held mail report goes to each user. This leads to one of the shortcomings of the service: If a user discovers a false positive, he or she must request whitelisting. Each request goes to the exception requests filter for the administrator to review. If approved, the requests are added to the whitelist. This means that the admin must review each and every whitelist request separately. If you take the 94 false positives I got during the first two weeks of testing and multiply it by several hundred users, you're looking at an inordinate amount of the admin's time during the first few weeks.
Both the admin interface (screenshot) and user interface are clean and easy to use, with drop-down boxes rather than text boxes for specific entries. The administrator can copy messages, bounce them, add an identifier to the subject line, delete, forward, allow, or hold (quarantine) spam. The service allows admins to set policies for individual users, but not the creation of groups.
The Web console used for setup and maintenance does have one annoying feature: If you leave it open, it times out after a short while. When you click on a link, you don't get a message saying the console has timed out, nor do you get a log-in box. You just get a message: "Login Failed: Username not found."
Pricing for SecureTide starts at $1.50 per mailbox per month and includes 24/7 U.S.-based support. The first month is free. Emergency E-mail Service (EMS) provides a backup e-mail server, either POP3/IMAP or Exchange Hosted Service on demand if your on-site server fails, for an extra 50 cents per user per month.
MessageLabs E-mail Anti-Spam and Anti-Virus Services
The MessageLabs service had the best performance in my testing, catching 97 percent of incoming spam with only eight false positives and zero critical false positives. It also boasts a stellar feature set, including anti-virus, control over images received, excellent administrative controls, a self-service portal, superior auditing tools, and the best LDAP synchronization software of the test. All of this comes at a price: The service runs $2.16 per user per month at 1,000 users, a little more expensive than the others in the roundup. Notably, MessageLabs was acquired by Symantec during the course of this test, though the brand will be retained and the service will remain the same.
Setting up the MessageLabs service is generally quite straightforward. The Active Directory/LDAP synchronization software works easily without requiring IT to open a hole in the firewall or export files to a comma-delimited format. However, the log-in creates a random user name, not username@domain or anything memorable. In my case, it was MED8559, and neither IE nor Firefox recognized or remembered the log-in. The unalterable password policy is also incredibly irritating: Passwords must contain capitals, lowercase, numbers, and symbols, which makes them difficult to type and nearly impossible to remember. Thus, many users will either write down log-ins and passwords on yellow stickies or call IT because they've forgotten them.
These unalterable security measures are annoying, especially in contrast to the service's policy engine, which is flexible in every way. Admins (screenshot) can, for instance, devise policies and settings by time of day, group, user, and more.
In addition to best-of-class anti-spam performance (see test results), MessageLabs offers a porn detection capability, scanning images for "excessive" bare skin, number of people, and such.
Additionally, images of photos or documents can be uploaded and used to create a signature to ensure that specific intellectual property is not sent or received without authorization. For example, if you have a confidential document titled ProprietaryInfo.doc, you can create a signature of that file that records not only the name by its length, content, and so on. MessageLabs will detect and stop any attempt to send that document to an outside recipient.
Although the MessageLabs service is relatively expensive, it also offers the most features and best performance among its rivals in this test. Now that the company is part of Symantec, which makes the Brightmail appliance, the consistent top performer in appliance tests over the past few years, maybe some of the irritating foibles will be fixed.
Microsoft Exchange Hosted Filtering Services
Formerly Frontbridge (long a leader in anti-spam services), Exchange Hosted Filtering Services (EHFS) is available in typical Microsoft fashion: The service offers great features, including anti-virus, using multiple engines, encryption, and disaster recovery. It's adept at stopping spam, preventing 97.5 percent of unwanted messages from coming through, while stumbling on 11 false positives and one critical false positive (see test results). Moreover, it's easy to set up, manage, and use -- especially with Windows-based e-mail systems. On the other hand, pricing is hard to nail down, though large customers get great price breaks. Also, as one might expect from Microsoft, using the service with the Firefox 3.0 browser did not work well, although Firefox 2.0 was OK.
Getting started with EHFS was quite easy: The process of setting up users can be automated so that administrator intervention is not required. Syncing with Active Directory was a snap -- and synchronization withversions of LDAP should work well too.
User log-in is straightforward. Passwords can be set to high security levels, requiring upper- and lowercase letters, numbers, and symbols, as well as a minimum length. You can even require users to choose password that are in any dictionary.
Notifications sent to users regarding quarantined e-mails are easy to read. Moreover, the user portal is simple and direct, which should minimize user support requirements.
Admins will find (screenshot) the service's policy engine to be powerful and easy to use, offering a high level of granularity for both individual users and groups. Additionally, EHFS offers potent content-filtering features for enforcing HR policies and controlling the distribution of intellectual property. You can set the content filter to look for specific file names or document types. You can even scan attached documents for specific phrases. Admins can choose to have the service block a user's attempt to send a document containing restricted content; alternatively, an admin could have the system notify a designated auditor of such attempts.
The service's reporting and auditing features are both powerful and well thought out. Creating customized reports is a snap, so you needn't rely on pre-defined reports if they don't meet your requirements. Additional features such as encryption and disaster recovery are easy to use and well integrated.
Pricing for Exchange Hosted Filtering Services starts at $1.75 per user per month for a small business (with a minimum of five users). Pricing may be much less in large volumes (there are no published fees). As a bonus, many Microsoft customers may be able to use Exchange Hosted Filtering at no additional charge through the Exchange Enterprise CAL, Microsoft Enterprise CAL, or Forefront Security Suite.
With excellent performance and usability, as well as pricing that may be included for Microsoft enterprise customers, Exchange Hosted Filtering Services offers a lot of capability at a price that is pretty good -- probably.
MX Logic Email Defense Service
Geared toward large enterprises, MX Logic's Email Defense Service (EDS) offers a nice array of features, including anti-spam, anti-virus, content management, and e-mail continuity, all at a very low price. Unfortunately, its spam-stopping abilities, though acceptable at 95 percent, were near the bottom of the barrel in this roundup. I found a relatively high number of false positives as well -- 96, along with four criticals -- meaning users who receive a high number of legitimate bulk e-mail will need to whitelist a lot of messages (see test results).
Setup was trouble-free, with automatic provisioning of users and easy access to settings on a per-user, per-group, or per-domain basis.
The end-user UI is also well designed, suited for even the most unsophisticated user. Users receive spam reports listing quarantined messages, along with links for automatically logging in to the user portal and allowing whitelisting or blacklisting easily.
Among EDS's extensive feature set is e-mail continuity. If the local mail server fails, users can view undelivered messages via a Webmail interface.
MX Logic's policy engine is among the most granular among the services I tested, and relatively easy to use. (MessageLabs fared slightly better in this area.) Policies can be tailored to flag both incoming and outgoing messages for racially insensitive terms, obscenity, or sexual overtones. Moreover, the engine can look for attachments, Java, or other potential exploits, as well as URLs or other links in messages. Reporting defaults to month-to-month (screenshot), but adding custom reports is easy.
Pricing for MX Logic EDS is good at higher volumes: $1.60 per user per month for 1,000 users (or $2 with message continuity service) and 99 cents per user per month for 5,000 users ($1.26 with message continuity). There are no setup fees, and 24/7 live telephone and Internet support is included. Online training for users and admins is available every week, and online or phone help is based in the United States.
Thanks to MX Logic EDS's excellent feature set and very low pricing at high volumes, the service's less-than-stellar spam-filtering performance is easy to overlook. If your users don't receive a lot of bulk e-mail or are willing to spend a couple of weeks whitelisting messages, they'll probably find performance perfectly satisfactory.
SECNAP Hosted Email Security Gateway
Although not without its strengths, SECNAP came away with the lowest overall score in this roundup. Its general spam-stopping capabilities were just fine at 97.5 percent; however, it had more false positives than any other service in this test -- 133, along with 10 criticals -- a problem that persisted throughout my review (see test results). On top of that, I was often frustrated with both the user and management interfaces.
Its other features are comparable to the rest, with anti-virus and content management. It doesn't offer built-in word lists for content management, but does include an encryption capability.
If you set up accounts through the service, you'll find that SECNAP requires users to create a strong password that includes uppercase and lowercase characters, as well as numbers and symbols. Once again, this practically ensures that users will write down their passwords somewhere (or call the help desk frequently). If, on the other hand, you create accounts using directory synchronization, you'll find a single sign-on feature allowing users to log in to the service with their standard Windows domain credentials.
SECNAP has a trait that admins will likely find quite frustrating: Whenever you make or accept changes to settings, the screen will take a long moment to refresh, sometimes twice. Also irritating: Accessing a quarantine directly is impossible. Rather, admins have to manually log in from another computer (because log-in info is cached) before they can get to the standard user-level quarantine access. Alternatively, you have to generate a quarantine report, send it to yourself via e-mail, open it, click on the embedded Web link, and then enter the user log-in embedded in the message before you get to the actual quarantine. In every other appliance and service I've tested, admins can click on a menu item from within the admin UI to access the quarantine.
Also, it would be nice if there was a way for an administrator to mark spam with a standard addition to the subject line and then set up a rule on the company e-mail server to direct those marked messages to users' spam folders. As it stands, users will have to access the service's Web site to deal with those messages.
Users will also suffer some annoyances with the SECNAP interface. After marking and releasing a message from quarantine, users will be taken to the first page of the quarantine. Additionally, when a user releases a message from quarantine, it still shows up in the quarantine with no indication of success.
Reducing false positives proved troublesome in my tests. Unlike with all the other services, the average number of daily false positives did not drop off over the course of the test. There were several reasons for this: First, when you click on an item, you can release it and whitelist the sender, but only as the exact sender's e-mail address (e.g. firstname.lastname@example.org); there's no way to whitelist an entire domain or add wildcards to an address before whitelisting. This means that whitelisting bulk e-mail from a sender that adds a random character string to each message is impossible.
Second, some odd default rules in the anti-spam policies can produce false positives. For example, RFC standards forbid using eight-bit subject lines in headers. Thus, if a message's subject line contains even a single eight-bit figure, such as foreign symbols, the registered character, or the trademark character, the message ends up in quarantine -- even if there's nothing else odd about it. This can be defeated, but only if you know what to look for -- and there's no obvious reason to change a setting called "header checks disabled."
Third, there were some disparities between the "apparently from" domain and the actual sender. For example, a message may appear to be from email@example.com, but the actual sender is firstname.lastname@example.org. (This is common practice for bulk e-mailers.) However, there's no way for the user to see or whitelist the actual sender, and whitelisting the "apparently from" sender doesn't let messages through.
Finally, when the high number of false positives continued to be an issue, I disabled Sender Policy Framework checking (SPF is a standard intended to help identify illegitimate e-mail). Not only were a lot of legitimate marketing messages still being stopped, but there were a number of critical false positives afterwards as well.
To SECNAP's credit, admins are provided with a high degree of granularity. You can assign limited rights to a lower-level admin to look up, release and query e-mail records, set default domain policies, and whitelist and blacklist messages. You can also permit users to log on and look at their e-mail logs, reports (screenshot), and quarantine (if enabled), set their own policies, and whitelist and blacklist messages.
The service also offers multidomain support. You can create different administrator accounts and policies for different domains (company1.com, company2.com, etc.) Additionally, companies can set up virtual domains ending with ".net," ".info," ."corp," and the like, even if their "real" domain is ".com." Thus, messages sent to email@example.com or firstname.lastname@example.org will still go to the intended recipient at email@example.com. All policies would be created on the real domain.
Pricing for SECNAP is very reasonable at $1 per user per month for 1,000 users. However, despite the service's reasonable cost, as well as its nice multidomain support and good administrative features, the ongoing problems I experienced with false positives make it difficult to recommend SECNAP to organizations where users need to be able to receive much bulk e-mail.
Trend Micro InterScan Messaging Hosted Security
InterScan Messaging Hosted Security (IMHS) from Trend Micro demonstrated excellent spam-busting performance in my testing: It stopped 97 percent of the incoming spam, with just 12 false positives and one critical, landing it in third place by a very small margin. Additionally, false positives and critical false positives were very low (see test results). Furthermore, Trend Micro offers basic features such as anti-virus. Advanced features include content management, for both enforcing HR policies and protecting intellectual property. The icing on the cake here is the pricing: At 1,000 users, it runs $1.06 per user per month for the basic feature set and $1.60 per for the advanced feature, making InterScan the least expensive offering in this roundup. Even better, Trend Micro's is the only service that comes with an SLA.
Setting up the service is simple. Provisioning users through the portal is fairly easy, too: Each users fills out a simple form the first time he or she accesses the site, after which his or her account is provisioned automatically. Admins can import user information from Active Directory to verify that e-mail is being sent to legitimate recipients. However, there is no way to sync the directory information or use the AD information to provision accounts.
Once user accounts are set up, the user experience is a breeze. The reports on quarantined messages (screenshot) are user-friendly, and whitelistingmessages is fuss-free.
Establishing policies is simple, and admins can easily create powerful policies to filter incoming and outgoing messages based on specific words, attachment types, or other criteria. Admins can even create specific policies to block or quarantine messages containing encrypted zip files. It doesn't include the signature capability that MessageLabs has, which prevents users from sending restricted documents, even if their names have been changed.
Reporting tools for the admin are good, and reports are easy to read, should it become necessary to show executives how well the service is performing.
All in all, IMHS provides great performance and a very fine feature set at a very low price and with good functionality. It lacks the granularity and ease of use that you'll find in MessageLabs, but it's also much less expensive.
Choosing the right service for your organization can depend on a number of factors, including false positive rate, feature set, and price. If price is the most critical consideration, Trend Micro InterScan Messaging Hosted Security is a great choice. If a very capable, flexible feature set is your first priority, MessageLabs may be your best bet.
The user experience -- both for admins and end-users -- is another key differentiator among all of these services. The only real way to find out which services are most appealing to you and your users is to test-drive your top choices. Fortunately, all of the providers offer free trials. Trying out a service entails simply changing the DNS records for your domain to point at the hosting site, and the trial period begins. You can try the service for at least 30 days to find out how it works with your mix of e-mail and how your users like it.
Overall Score (100%)
|MessageLabs Email Anti-Spam and Anti-Virus Services||9.0||9.0||9.0||9.0||8.0|
|Microsoft Exchange Hosted Filtering Services||8.0||9.0||9.0||9.0||8.0|
|MX Logic Email Defense Service||8.0||8.0||9.0||8.0||8.0|
|SECNAP Hosted Email Security Gateway||7.0||7.0||8.0||7.0||8.0|
|Trend Micro InterScan Messaging Hosted Security||8.0||9.0||8.0||7.0||9.0|
Windows 7 is suddenly telling users it isn't genuine -- and it has nothing to do with Windows being...
Windows users are reporting significant problems with four more October Black Tuesday patches
The larger design is very welcome, but there's much more to the iPhone 6 than a bigger screen
Sponsored by Rackspace
Sponsored by Nuage Networks
Sponsored by Fibre Channel Industry Association
InfoWorld picks the best hardware, software, development tools, and cloud services of the year
Microsoft CEO Satya Nadella is showing the same kind of leadership that Steve Jobs used to rescue Apple...
If you’re doing one or more of these things, it might be time to step away from the IDE and take a...
Black Duck presents its Open Source Rookies of the Year -- the 10 most exciting, active new projects...