Commercial solutions win, spam loses

Brightmail, FrontBridge, Postini, and Proofpoint overwhelm open source in accuracy, flexibility, and ease

As anyone with an e-mail inbox knows, the spam problem isn't going away. According to a major anti-spam vendor, spam has increased from 8 percent of all e-mail traffic in 2001 to 50 percent in July 2003. Other estimates show that figure as high as 70 percent of all traffic. Two classes of products can help slay spam in the enterprise environment: gateways and services. Both allow you to block spam for all network users at a single, centrally managed point before it hits your mail server.

For this review, I looked at two services and three gateway products. Services filter spam before it arrives at your network, reducing the volume of traffic on your Internet connection. Services also typically offer multiple datacenters for redundancy, high volume, and fast response. Setup requires merely changing the MX (mail exchange) record for your domain. But a service is not under a local administrator's control, so if the service goes down, mail may not get through.

Gateways are harder for spammers to circumvent by sending e-mail to the real mail server's IP address; they offer local control of the anti-spam technology; and they allow mail to continue to arrive if the anti-spam gateway goes down. But a gateway gives the local administrator yet another system to maintain, and the total traffic through your Internet connection remains the same because spam isn't filtered until it reaches your network.

The five products I tested: Brightmail Anti-Spam Enterprise Edition Version 5.1, FrontBridge TrueProtect E-mail Security Suite, Postini Perimeter Manager Enterprise Edition, Proofpoint Protection Server 1.2.1, and SpamAssassin 2.44, an open source spam filter included with Red Hat Linux 9.

In contrast to the commercial products, SpamAssassin represents an older, first-generation anti-spam solution, and its age showed in my tests. It filtered only 62 percent of spam, whereas the other products produced great results, blocking 90 percent to 96 percent of all the spam they encountered with few, if any, legitimate messages blocked.

Differentiating between spam and legitimate messages can be difficult. Newsletters, press releases, and other marketing materials from companies you have a relationship with can be very similar to spam in content. These all present challenges to the filters. The e-mail I used for testing was real e-mail containing many messages that stressed the filters.

I looked at two categories of mail incorrectly identified as spam: false positives that were not critical, such as newsletters and marketing information; and false positives that were critical, such as personal e-mail from colleagues. Each product was tested with a different stream of mail, so the number of messages received varied, but all received enough messages to assess their capabilities.

The critical issue is not that the filter may have misidentified a few e-mails, but how easily those messages can be found and added to a whitelist so that future e-mails from the same source are not stopped. All the products except Brightmail and SpamAssassin allow end-users to add senders to the domain whitelist themselves. Brightmail allows users to forward misidentified e-mails to the administrator, who can choose to add the sender to the whitelist. SpamAssassin allows only the administrator to add to the whitelist, with no direct access for users.

All the products allow the administrator to blacklist known spammers and choose among a variety of responses to messages identified as spam -- adding an identifier to the subject line, adding a message header, deleting the message, or quarantining it. Delegation of specific administrative functions is possible with all the products except SpamAssassin, although the granularity of delegation varies among the four. Spam settings can be set by enterprise (multiple domains) or domain, and Postini also allows individual groups or users within a domain to have different rules.

And all the products but SpamAssassin use dynamic updates to keep up with the evolving technologies spammers use to circumvent less sophisticated filters. The default update cycle may be every few minutes or once per week, depending on the product. Keeping the filters up to date requires a subscription or maintenance fee.

Finally, in addition to stopping spam, all four commercial products provide content-filtering features, allowing the administrator to block incoming or outgoing e-mail that contains proprietary data, audio or video files, executables, sexually explicit words, or racial slurs. They also provide protection against DoS attacks and directory harvesting attacks.

In my testing, the performance of the newer products was more than acceptable in every case. Per-user, per-year pricing should not be an obstacle, even for the most expensive product. Choosing the right product will depend on your network topology, your philosophy regarding outsourcing, requirements for administrative control and reporting, traffic loads, and your operating system and mail server platform.

Click for larger view.

Brightmail Anti-Spam Enterprise

This gateway product constantly interacts with Brightmail's datacenter to keep filtering rules current. The gateway polls Brightmail's datacenter every few minutes and downloads new rule sets when they're available, in much the same way anti-virus applications do.

Brightmail's software can be installed on Linux, Solaris, or Windows, and features an easy to use GUI installer on all three platforms. I installed the gateway on a Windows 2000 server with Exchange Server 2000 and enabled Brightmail's Exchange spam folder agent in less than 10 minutes. The software automatically contacted the Brightmail site and downloaded the latest set of rules. No additional configuration or tuning was necessary. Brightmail caught the highest percentage of spam and had the lowest false-positive rate of any of the products tested.

Brightmail is the only product that does not allow end-users to add senders to the whitelist. On the other hand, Brightmail includes a spam folder agent for both Exchange and Lotus Domino -- all mail identified as spam can be sent to the end-user's spam folder, and an Outlook agent allows users to forward e-mail to the administrator, indicating "spam" or "not spam" with one click.

This makes scanning and recovery of false positives very simple and straightforward. Alternatively, mail identified as spam can be tagged as such in the header or subject line, and spam can be sent to a central spam mailbox, saved to disk, delivered normally, or simply deleted. You can configure different policies for different domains.

Brightmail offers extensive reporting features, a wide variety of standard reports as well as custom reports. An optional anti-virus capability powered by Symantec is available at additional cost, with virus definitions and engine updates delivered by Brightmail.

FrontBridge TrueProtect

FrontBridge is a hosted service that incorporates four layers of e-mail filtering -- custom blacklists, proprietary fingerprinting, adaptive rules-based scoring, and real-time attack prevention, which blocks illegitimate and potentially damaging e-mail based on a sender's IP address. FrontBridge was recently selected by Sprint as its anti-spam solution.

Installing FrontBridge consists of merely changing the MX record for your e-mail server to point to the FrontBridge mail processor. FrontBridge processes all your e-mail, incoming and outgoing, and forwards the good stuff to your mail server or its outbound destination. There is no impact on your local network configuration, and overall Internet traffic is reduced because spam never reaches your network. FrontBridge claims never to have had a service outage and guarantees 99.99 percent uptime. With eight datacenters worldwide, the company seems to have the infrastructure to make such a guarantee. FrontBridge offers additional services beyond anti-spam, including anti-virus, content filtering, policy enforcement (such as who can send and receive which file types), and disaster recovery, which involves holding all e-mail for as long as five days if your network is unreachable.

Configuring accounts and other administrative tasks is done through an HTTPS log-in to FrontBridge's Web site. Setting up accounts is simple: An automated user enrollment feature allows all the accounts in a domain to be added without having to build an access control list. Administrative tasks, such as modifying filter rules or anti-virus settings or adding and deleting users, can be set by domain so that each of several domains or sub domains can each be maintained by different administrators.

Reporting is excellent, and reports can be easily exported to Excel for analysis. By default, a digest of filtered spam is delivered weekly to all users as an HTML e-mail. Users can retrieve any e-mail that has been quarantined, and can whitelist the sender with a single click. End-users can also log in to the Web site at any time and view all filtered messages with the same options to deliver the message or whitelist the sender.

FrontBridge caught 90 percent of the spam in the test, ranking below Brightmail, Proofpoint, and Postini in accuracy. But it misidentified no critical e-mail, and only 1 percent of noncritical messages, proving more adept than all but Brightmail at avoiding false positives.

Postini Perimeter Manager

Click for larger view.

Postini's anti-spam service processes about 150 million messages per day. Although it started as a service for ISPs, it has recently moved into the enterprise space and provides a broad, sophisticated array of services. It is the only product I tested that includes anti-virus scanning in the base price.

Setting up the service is simple, requiring the same MX record change as FrontBridge's service. Adding users is automated and very easy -- each user receives a message the first time that spam is blocked from their account, letting them know how to access quarantined e-mail and retrieve, delete, or whitelist mail. All administrative tasks can be accomplished through the Postini Web site, and management tasks can be delegated in a very granular manner. Managing multiple domains is easy. Reporting is flexible in the criteria reported, but long-term tracking is not available in the standard corporate edition -- only daily and weekly reports are made available.

Response to spam is unusually flexible, and can be set by individual, group, or domain. Administrators can allow users to add senders to the whitelist, retrieve messages from quarantine, and even change filter settings -- or they can lock things down so that end-users can do nothing without an administrator. The spam filters have separate settings from lenient to strict for a variety of categories, including bulk e-mail, special offers, get-rich-quick messages, and adult content.

The Standard Edition includes spam filtering, inbound server monitoring, connection management, delivery management, detailed reporting, inbound attachment management, inbound virus blocking, and inbound content management. The Enterprise Edition adds outbound server monitoring, outbound virus blocking, outbound attachment management, outbound content management, and disaster-recovery service. It can also check outbound e-mail for policy violations concerning language, recipients, and attachments.

Postini is very flexible and feature-rich, and it caught nearly 94 percent of spam in my tests, edged out only by Brightmail and Proofpoint. It lagged slightly in avoiding false positives, but the differences here could easily be overcome by whitelist tuning.

Proofpoint Protection Server

The Proofpoint Protection Server is a gateway that runs on Linux (Red Hat 8 or 9) or Solaris. Enterprises using Solaris or Linux and sendmail will find it a comfortable, easy fit. Fortunately, companies using Exchange, Notes, or other e-mail platforms can rely on Proofpoint to get things running. Proofpoint will even install its server on a system you send to it at no additional cost.

I installed the software on Red Hat Linux 9, with help from one of Proofpoint's systems engineers. She talked me through getting the Linux system configured properly, getting sendmail set up, and installing and configuring the Protection Server, which includes the MySQL database server for storing quarantined e-mail.

Configuration is simple, and delegation is straightforward -- although not as granular as it is in Postini. Multiple administrators can be created, and each has a limited set of seven areas to which they either do or don't have access. Rather than the two categories the others use in their reports, "spam" and "not spam," Proofpoint has three: "definitely spam," with a score of 80 to 100; "probably spam," with a score of 50 to 80; and "definitely not spam," with a score of 0 to 50. The qualifying scores can be changed for each category, and the action taken on the message can be different for each. For example, you could opt to delete messages that fall into the "definite spam" category and quarantine those in the "probable spam" category. Content filtering is also easy to set up, with a dictionary of undesirable terms included.

As often as administrators like, clients are sent a digest via e-mail that allows them to view quarantined e-mail, sorted by likelihood that it is spam. Users can release e-mails from quarantine and can whitelist senders directly from the e-mail client.

Proofpoint was second only to Brightmail in accuracy, catching more than 94 percent of spam. It also had no critical false positives, although its ability to recognize legitimate mass mailings fell slightly short of the three other commercial products.


You get what you pay for. SpamAssassin is an open source gateway that is included with Red Hat Linux 9, and can be downloaded free from However, it took more than 10 times as long to install and configure SpamAssassin as it did any of the other products. I achieved a much lower level of performance to boot -- roughly 63 percent accuracy in identifying spam, with a relatively high number of false positives.

I installed SpamAssassin Version 2.44 along with Red Hat Linux 9. Installing Red Hat 9 is easy, and the SpamAssassin package is included with the mail server installation. But just because the software is installed does not mean it will work -- filtering criteria must be added manually, and until that's done nothing is filtered out. Getting the various configuration files edited properly so that the whole package worked was not simple. Documentation was difficult to find, and not always easy to follow.

There are blacklists available that you can subscribe to, and some are updated regularly, but these are noncommercial lists with no guarantees. The whitelist is not difficult to add to, but there is no mechanism for end-users to add to the whitelist or to automatically notify the administrator to add senders. Filtering rules are relatively basic, and although there is a Bayesian filter available, it is not part of the distribution -- and I wasn't able to get it working for this review.

SpamAssassin is the perfect example of first-generation techniques becoming outmoded by advances in spamming technology. It looks for keywords in the subject or body of e-mails, but is frustrated by words not in the dictionary, such as "V!agra," or words that contain invisible HTML characters. It might be possible to get SpamAssassin to perform at a level similar to the other products reviewed here, but it would take a lot of work in addition to constant maintenance and research by the administrator.

Choosing Your Weapon

All of the commercial products worked well in my tests, and all should prove satisfactory in a corporate environment. After initial setup and a few weeks of tuning the whitelists for your organization's mail stream, false positives should drop to very near zero, and any of these anti-spam solutions should disappear into the background, requiring little attention.

The two services, FrontBridge and Postini, receive higher marks for setup and management -- they are easier to install and administer than the gateway solutions. The biggest drawback to services (e-mail won't get through if the service goes down) shouldn't be a factor with either of these solutions, as both have multiple datacenters and excellent reliability records.

FrontBridge offers a good feature set and is very easy to use, with excellent reports and fine anti-spam performance. Postini has the broadest feature set of any of the products I looked at, including the greatest range of controls over filter settings by user, group, or domain, and by types of spam filtered. Its controls over content filtering on inbound and outbound messages are also the most complete, an advantage for managers concerned about liability for e-mail content.

The gateways may make some paranoid (read: experienced) administrators more comfortable because they're nearly impossible to bypass by targeted spam attacks and they're completely under local control. They all require subscriptions or maintenance fees to keep working, so there isn't much difference from a service in that respect. Brightmail offers the broadest platform support among the gateways, nice integration with Exchange, and great accuracy in identifying spam -- the highest in the test. It was also the most appliancelike in installation and setup -- a real "set and forget" system.

Proofpoint is a good choice for Linux or Solaris shops, providing extremely high accuracy, great support, and excellent manageability through a Web interface. The next version, due in early December, will make filtering options by users and groups available, as well as provide additional reporting tools and management features.

Considering the price per user, per year, and given the time they'll save your users and administrators in dealing with spam, they're all bargains.

InfoWorld Scorecard
Ease of use (20.0%)
Manageability (25.0%)
Value (10.0%)
Accuracy (25.0%)
Setup (20.0%)
Overall Score (100%)
Proofpoint Protection Server 1.2.1 8.0 8.0 8.0 9.0 8.0 8.3
SpamAssassin 2.44 6.0 7.0 6.0 5.0 6.0 6.0
Brightmail Anti-Spam Enterprise Edition 5.1 8.0 8.0 9.0 9.0 8.0 8.4
Postini Perimeter Manager Enterprise Edition 9.0 9.0 8.0 9.0 9.0 8.9
FrontBridge TrueProtect E-mail Security Suite 8.0 8.0 8.0 9.0 9.0 8.5
From CIO: 8 Free Online Courses to Grow Your Tech Skills
You Might Like
Join the discussion
Be the first to comment on this article. Our Commenting Policies