The truth about anti-phishing toolbars

Carnegie Mellon's CyLab studies anti-phishing toolbars. Conclusion: They all stink.

In recent weeks, there's been a kind of 'battle of the surveys' over the thorny issue of which anti-phishing technology is best. The scrum started back in September, when the folks behind IE7 took the opportunity to blog about the findings of a Microsoft-sponsored study of anti-phishing toolbars. That study, conducted by 3Sharp, compared anti-phishing toolbars from Microsoft, NetCraft, Google/Firefox, AOL, EarthLink, eBay, Geotrust, Netscape and McAfee and used 100 known phishing Web site URLs and 500 known good URLs to see how well each anti-phishing technology flagged both phish and legitimate URLs.

Not surprisingly, Microsoft's Phishing Filter (MPF) in IE 7 Beta 3 received the highest "composite score" followed closely by NetCraft's toolbar.

The folks over at Mozilla responded a couple of weeks ago, releasing the results of a study by "independent firm" SmartWare showing that -- no -- it's Mozilla Firefox 2.0 anti-phishing filter beat IE7, thus showing Microsoft that they know how to play the "sponsored study" game too.

As both the Microsoft and Mozilla studies indicate, it's easy to tweak your study in a way that benefits the particular anti-phishing technology you want to come out on top. That was clearly the case with the IE7 study, where IE7's toolbar didn't do the best job of spotting phish sites, but won on points using a scoring system in which blocking sites was what garnered the most points. And in each case, the sample phishing sites are more or less an unknown quantity that could, potentially, aid one type of anti-phishing tool over another.

What's been needed all along, of course, is a truly independent and scientific study of anti-phishing toolbars, and that's what we now have, courtesy of Carnegie Mellon University, which released a comprehensive study of ten toolbars. According to a report released by CMU's CyLab, none of them are all that great.

The study, which is dated Nov. 13, looked at anti-phishing toolbars from eBay, Earthlink's, GeoTrust's, Google/Firefox, McAfee (SiteAdvisor) and Microsoft (IE7), NetCraft, NetScape and Stanford University (SpoofGuard). The researchers, working with nothing more than government funding, conducted two experiments assessing the effectiveness of the toolbars using an automated test bed 100 phishing URLs and 100 unique domains from the Anti-Phishing WOrking Group. They found that just three of the 10 toolbars: SpoofGuard, EarthLink and Netcraft, were consistently able to identify over 75% of the phishing sites tested (Google's got that good, but only after the site had been live for a couple hours. IE7 hovered just below 70 percent, as did CloudMark.) But of those three top performers, SpoofGuard misidentified 38 percent of legitimate sites as phishing sites which, as the researcher spoint out, tends to undermine the value of SpoofGuard overall.

Four of the toolbars tested (GeoTrust TrustWatch, eBay, Netscape and McAfee SiteAdvisor) were not able to identify even half the phishing sites tested, CyLab reported.

From the report:

"Overall, we found that the anti-phishing toolbars that were examined in this study left a lot to be desired. SpoofGuard did a very good job at identifying fraudulent sites, but it also incorrectly identified a large fraction of legitimate sites as fraudulent. EarthLink, Google, Netcraft, Cloudmark, and IE7 identified most fraudulent sites correctly and had few, if any, false positives, but they still missed more than 15% of fraudulent sites. The other four toolbars we tested could correctly identify less than half the fraudulent sites, and one [McAfee SiteAdvisor] did not correctly identify any fraudulent sites."

Adding insult to injury (for users, anyway), CyLab found that many of the toolbars they tested were vulnerable to exploits such as CDN (Content Distribution Network) attacks the mask the true URL used in an attack. In the end, the CMU/CyLab researchers were left wondering how useful these anti-phishing toolbars even are.

"Even if it is possible to create a technically sound antiphishing toolbar, it is still unclear as to whether or not this would be beneficial to users. Usability problems plague all varieties of software, security software in particular. When using an anti-phishing toolbar, poor usability could mean the difference between correctly steering someone away from a phishing site and having them ignore the warnings only to become a victim of identity theft."

Suggested study topic for CyLab: IDS/IPS!


Copyright © 2006 IDG Communications, Inc.

How to choose a low-code development platform