How I tested

Testing anti-spam products is a challenging task. Collecting a large variety of spam and forwarding it through the gateway is a simple way to test, but it makes the test much less effective, because most products look for the sender and the sender’s IP address as major clues as to whether or not the message is spam.

It’s important to have real mail coming in, both personal messages and mailing lists, which many products have a hard time distinguishing from spam. I used four separate e-mail accounts for testing, each receiving a mix of personal messages, e-mail newsletters, messages from PR people regarding new products, and spam.

The mix of messages was a difficult one for anti-spam filters. For example, I receive a lot of press releases by e-mail. The characteristics of these messages are similar in many ways to marketing spam, which makes it hard for filters to distinguish among them, both because of the verbiage and the fact that they are often distributed by bulk e-mailers. Likewise, newsletters, technical ones such as those offered by InfoWorldand opt-in marketing information or product update information can trigger the filters. Personal e-mail can also trigger the filters, because some of them look for typical spammer e-mail addresses, which often contain a group of characters followed by numbers, a characteristic also true of many personal accounts on AOL, MSN, Juno, and other large providers. Friends or family who often send pictures of the kids or use cute HTML e-mail backgrounds may also have their e-mail filtered.

Because I normally use hosted e-mail, and download it with POP, I had to make some changes for this test, which required SMTP for the gateway products. I created an SMTP-forwarding server and used a POP downloader to retrieve mail and send it via SMTP to my Exchange server. This gave me a stream of SMTP traffic to filter for the gateway products.