Test Center: Sandbox security versus the evil Web

The Internet is a scary place. Criminal malware lurks on legitimate and illegitimate Web sites alike, looking to steal your money one way or the other. Vendors have been scratching their collective heads attempting to make more consumers safer, more often. One of the results has been a class of anti-malware software that I call sandbox protection products. These items encapsulate Internet browsers (and e-mail programs and sometimes any other program you can run) within a virtual, emulated cocoon designed to keep malware from reaching and modifying the underlying host computer.

It used to be that you had to boot with an infected floppy diskette, run an infected executable, or double-click on an e-mail attachment to get exploited. Nowadays, all you have do is surf your browser across the wrong Web page, or the right Web page at the wrong time. Client-side, polymorphic, Internet browser-based exploits account for the large majority of malware infections. And although nearly ubiquitous in use, conventional anti-virus, anti-spam, and host-based firewalls are being challenged as never before to provide protection. In the virtual world, malware can be controlled, limited, and defanged. If the vendor does its job perfectly, everything legitimate the end-user wants to keep is kept permanently, and all traces of malware are erased as if the exploit never occurred.

[ For the back story to this review, see "Does sandbox security really protect your desktop?" and Roger Grimes' original review of ZoneAlarm ForceField as well as his discussion of red/green computer security in his Security Advisor blog. ]
As this review will show, none of the current solutions are perfect, but they do have value as an additional line of defense. I tested five products: Check Point Software's ZoneAlarm ForceField, Sandboxie, Prevx, Authentium's SafeCentral, and Softsphere Technologies' DefenseWall HIPS. Trusteer declined our invitation to participate, and one of the original pioneers in this field, GreenBorder Technologies, has been purchased by Google and is unavailable for review. I intentionally wanted to cover products that would be new to most of our readers and that have not been reviewed multiple times in the past.

Each product was tested by subjecting it (and the underlying host running Microsoft Windows XP Professional SP2, Internet Explorer 6, Firefox 1.0, and several older versions of browser add-on software) to hundreds of malicious Web links. Unpatched application software was intentionally used to test the defensive capabilities of the reviewed products. I didn't want the latest vendor patches stopping the malware. I wanted the sandbox products to do all the hard work.

I surfed to real-world Web sites that hosted computer viruses, worms, bots, Trojans, malicious scripts, and rogue batch files, then attacked using local and remote buffer overflows, all while logged on using the built-in Administrator account. I used malicious links provided by public and private anti-malware discussion lists and sources. I ran both old and new exploits, attempting to mimic the common threats the average user could encounter while surfing the Internet.

I was keen to see how well programs prevented silent "drive-by" downloads and how well they protected the user even if the user intentionally installed them (as if provoked by social engineering). Some sandbox products only provide protection from silent downloads, which can be furnished by a fully patched system without additional software in most cases. Others provide protection no matter what the user does, which is even more important in today's world of sophisticated social engineering.

Concerns about the class
But before reading the individual product reviews, let's discuss sandbox software in general. Sandbox protection products haven't gained a tremendous amount of traction with customers over the years for a number of legitimate reasons.

The first concern is accuracy. Every product failed one or more tests to varying degrees. All of them failed the Adobe Flash clipboard hijack exploit test, and most failed to accurately clean up from the XP Antivirus malware program (see the sidebar, "Two tenacious exploits debunk vendor claims"). This was despite the fact that many sandbox vendors claimed to prevent all known and unknown attacks. You can see the results and failures in the many screen images and video files offered along with this review.

The question is, despite the dubious accuracy, do these products provide additional value? In most cases, the answer is yes. Most sandbox programs attempt to prevent any system modification and don't care whether a particular threat is "recognizable." But this causes a tremendous amount of false negatives, meaning real threats aren't identified as such, and leads to a second problem.

Inherent in many of the products is the idea that end-users must make a trust decision on whether to erase, save, or execute downloaded content. Taken to one extreme, if end-users erase all content after every session, how would their system, applications, or browsers receive upgrades or security patches? Taken to the other extreme, if users save or execute all content, they will end up infected or negate the need for the additional protection. Ultimately, with varying levels of assistance from the product, the end-user must make the key decision on whether or not to save and execute the data from each session.

Detecting what is and isn't malicious is becoming harder all the time. A large majority of malware is coming from innocent, legitimate Web sites (such as favorite news sites, online social portals, blogs, and so on) that are infected with harmful content, and the social engineering pitches to the end-user are getting more persuasive.

Gone are the days when phishing malware was easy to spot due to obvious grammar issues and misspellings. Today's crimeware poses as legitimate vendor patches, online malware removers ("You are infected and need to run this scanner!"), overdue bills, and legal notices. Because of these increasingly blurred distinctions, end-users can't always be sure which Web site content can be trusted and safely executed. And still users are forced to make a trust decision that twenty years of history shows they aren't adept at making. If users could make consistently correct trust decisions, would they need the protection that sandbox products provide in the first place?

Some of the products in this review, notably Sandboxie and SafeCentral, rarely made an attempt to inform the user whether a Web site or download was legitimate or malicious. The user had to make every trust decision. Other products attempted to tell the user which Web sites contained malware and which did not. Prevx did a fairly good job at this, while DefenseWall and ForceField were more hit than miss.

In many products, content downloaded during a browser session must be saved or discarded as a whole (in other words, everything or nothing), while other products, especially Sandboxie and DefenseWall, allow the user to pick and choose between individual objects. I enjoyed the detail Sandboxie showed, as it often allowed me to confirm whether or not something malicious had occurred (such as new files in System32), but it really is only useful for technical users.

Sandboxie and DefenseWall focused on protecting particular applications or sessions, while others fell into the more traditional role of a host intrusion prevention system (HIPS), protecting critical system areas regardless of the attack vector. I was impressed with Sandboxie's ability to ensure that additional child sessions were always launched in protected mode when instantiated by a protected parent process. This is important as the browser is becoming more of a launching point for the rest of our integrated applications. Malware writers are increasingly attacking the applications as operating systems and browsers get more secure.

Nobody's perfect
Another important question is, how good is the emulation coverage? Sandbox protection products, by their very nature, don't emulate the entire operating system, as a full virtualization product such as VMware Workstation, Microsoft Virtual PC, or Parallels would. Malware programs are known to infect more than a hundred different Windows attributes, including registry locations, files, folders, startup areas, and more. How many Windows attributes and APIs are covered in the sandbox? The answer is never all. Does the product protect against remote and local buffer overflows, phishing attacks, alternative data stream techniques, file sharing avenues, and so on? Some did, most didn't.

Some of the products provided additional anti-buffer overflow, privacy, or phishing controls. The privacy and phishing controls are often already provided by other installed anti-malware programs, so their inclusion in this class of products may not be necessary (although additional layers of defense-in-depth never hurt).

Each product offered up differing levels of buffer overflow protection. For example, Sandboxie only prevented local buffer overflows if they happened against a protected process. Prevx protected the whole system against both local and remote buffer overflows, but only when they affected a critical system area being monitored.

Most of these products would not detect previously installed malware (Prevx being the exception) unless the malware made additional system modifications to the monitored areas after the products were installed. None of the products provided anti-DoS services, misconfiguration detection, missing patch analysis, or a host of other protections required to make a host system more fully secure.

Every product in this review worked only with Microsoft Windows. Some required Windows XP SP2 or later, although most worked with Windows 2000 and later versions. DefenseWall refused to defend Windows system processes. All worked with Internet Explorer and Firefox, although some of them would work with any program.

All of the products worked by installing one or more monitoring executables and services. Each provided a main executable and a system tray icon. Some of the tray icons changed colors, like a traffic light, to indicate current status (green for everything's OK to red for malware detected). All products displayed an on-screen warning when maliciousness was detected and most created log files. Interfaces ranged from Prevx's all-user elegance to Sandboxie's technical-user sophistication. The install, interface, and alerting for all products was acceptable. Pricing was $29.95 per copy or less.

Only Prevx had any enterprise capabilities, and even that was minimal. Most of the products were obviously intended for home or personal use. You won't find enterprise-wide reporting, logging, or alerting; or the capability to push out or monitor large-scale deployments. Sandbox defenses are first-generation products, sitting where anti-virus scanners were a decade ago.

Overall, this class of protection products does provide additional defense capabilities that could protect a user against unknown threats. In no case was using the vendor's product worthless, although some need to mature a bit to be ready for widespread use. The biggest question is if the additional protection value is worth the additional outlay of money and ongoing support. A fully patched system (OS and applications) where the user cannot install random programs would probably provide as much protection. How well your organization handles those two requirements will determine if sandbox products are worth investigating.

And the winners are…
In the end, the reviewer's favorite products were Prevx and Sandboxie. Prevx provided the best identification of malware and prevented most of the exploits thrown at it, though by no means all. It's nice to be told what was trying to infect your system instead of having to make trust decisions on the fly. Plus, Prevx was the only product able to detect previously installed malware, and its interface was elegant. Sandboxie was a surprise. It provided fairly accurate infection prevention and, in most cases, excellent cleanup. It requires a bit more technical knowledge when picking which changes should and shouldn't be kept, but it's free price tag makes it a winner.

Now on to the individual reviews…

Prevx 2.0
I've been a big fan of Prevx for years. It was one of the first players in the Web security space and tends to be on the cutting edge of browser defense. The product's maturity shows in the end-user interface, operational aspects, and availability in 64-bit and business versions.

Prevx provides a multipronged defense, with heavy reliance on heuristic host-intrusion detection techniques. It provides distinct protection to Internet browsers, e-mail programs, critical file and memory areas, and startup program areas, and it supplies additional defenses against keyloggers, buffer overflow programs, and network connecting malware. Although real-time monitoring and heuristics are certainly its sweet spot, Prevx contains multiple signature-based mechanisms and relies heavily on its community-based malware reporting database, which requires an active Internet connection to utilize.

[ View the companion video, "Prevx gets the slip." Download the QuickTime version. ]