Test Center: Sandbox security versus the evil Web

Five products strive to trap drive-by downloads and other threats in a virtual Web browsing space, with mixed results

The Internet is a scary place. Criminal malware lurks on legitimate and illegitimate Web sites alike, looking to steal your money one way or the other. Vendors have been scratching their collective heads attempting to make more consumers safer, more often. One of the results has been a class of anti-malware software that I call sandbox protection products. These items encapsulate Internet browsers (and e-mail programs and sometimes any other program you can run) within a virtual, emulated cocoon designed to keep malware from reaching and modifying the underlying host computer.

It used to be that you had to boot with an infected floppy diskette, run an infected executable, or double-click on an e-mail attachment to get exploited. Nowadays, all you have do is surf your browser across the wrong Web page, or the right Web page at the wrong time. Client-side, polymorphic, Internet browser-based exploits account for the large majority of malware infections. And although nearly ubiquitous in use, conventional anti-virus, anti-spam, and host-based firewalls are being challenged as never before to provide protection. In the virtual world, malware can be controlled, limited, and defanged. If the vendor does its job perfectly, everything legitimate the end-user wants to keep is kept permanently, and all traces of malware are erased as if the exploit never occurred.

[ For the back story to this review, see "Does sandbox security really protect your desktop?" and Roger Grimes' original review of ZoneAlarm ForceField as well as his discussion of red/green computer security in his Security Advisor blog. ]
As this review will show, none of the current solutions are perfect, but they do have value as an additional line of defense. I tested five products: Check Point Software's ZoneAlarm ForceField, Sandboxie, Prevx, Authentium's SafeCentral, and Softsphere Technologies' DefenseWall HIPS. Trusteer declined our invitation to participate, and one of the original pioneers in this field, GreenBorder Technologies, has been purchased by Google and is unavailable for review. I intentionally wanted to cover products that would be new to most of our readers and that have not been reviewed multiple times in the past.

Each product was tested by subjecting it (and the underlying host running Microsoft Windows XP Professional SP2, Internet Explorer 6, Firefox 1.0, and several older versions of browser add-on software) to hundreds of malicious Web links. Unpatched application software was intentionally used to test the defensive capabilities of the reviewed products. I didn't want the latest vendor patches stopping the malware. I wanted the sandbox products to do all the hard work.

I surfed to real-world Web sites that hosted computer viruses, worms, bots, Trojans, malicious scripts, and rogue batch files, then attacked using local and remote buffer overflows, all while logged on using the built-in Administrator account. I used malicious links provided by public and private anti-malware discussion lists and sources. I ran both old and new exploits, attempting to mimic the common threats the average user could encounter while surfing the Internet.

I was keen to see how well programs prevented silent "drive-by" downloads and how well they protected the user even if the user intentionally installed them (as if provoked by social engineering). Some sandbox products only provide protection from silent downloads, which can be furnished by a fully patched system without additional software in most cases. Others provide protection no matter what the user does, which is even more important in today's world of sophisticated social engineering.

Concerns about the class
But before reading the individual product reviews, let's discuss sandbox software in general. Sandbox protection products haven't gained a tremendous amount of traction with customers over the years for a number of legitimate reasons.

The first concern is accuracy. Every product failed one or more tests to varying degrees. All of them failed the Adobe Flash clipboard hijack exploit test, and most failed to accurately clean up from the XP Antivirus malware program (see the sidebar, "Two tenacious exploits debunk vendor claims"). This was despite the fact that many sandbox vendors claimed to prevent all known and unknown attacks. You can see the results and failures in the many screen images and video files offered along with this review.

The question is, despite the dubious accuracy, do these products provide additional value? In most cases, the answer is yes. Most sandbox programs attempt to prevent any system modification and don't care whether a particular threat is "recognizable." But this causes a tremendous amount of false negatives, meaning real threats aren't identified as such, and leads to a second problem.

Inherent in many of the products is the idea that end-users must make a trust decision on whether to erase, save, or execute downloaded content. Taken to one extreme, if end-users erase all content after every session, how would their system, applications, or browsers receive upgrades or security patches? Taken to the other extreme, if users save or execute all content, they will end up infected or negate the need for the additional protection. Ultimately, with varying levels of assistance from the product, the end-user must make the key decision on whether or not to save and execute the data from each session.

Detecting what is and isn't malicious is becoming harder all the time. A large majority of malware is coming from innocent, legitimate Web sites (such as favorite news sites, online social portals, blogs, and so on) that are infected with harmful content, and the social engineering pitches to the end-user are getting more persuasive.

Gone are the days when phishing malware was easy to spot due to obvious grammar issues and misspellings. Today's crimeware poses as legitimate vendor patches, online malware removers ("You are infected and need to run this scanner!"), overdue bills, and legal notices. Because of these increasingly blurred distinctions, end-users can't always be sure which Web site content can be trusted and safely executed. And still users are forced to make a trust decision that twenty years of history shows they aren't adept at making. If users could make consistently correct trust decisions, would they need the protection that sandbox products provide in the first place?

Some of the products in this review, notably Sandboxie and SafeCentral, rarely made an attempt to inform the user whether a Web site or download was legitimate or malicious. The user had to make every trust decision. Other products attempted to tell the user which Web sites contained malware and which did not. Prevx did a fairly good job at this, while DefenseWall and ForceField were more hit than miss.

In many products, content downloaded during a browser session must be saved or discarded as a whole (in other words, everything or nothing), while other products, especially Sandboxie and DefenseWall, allow the user to pick and choose between individual objects. I enjoyed the detail Sandboxie showed, as it often allowed me to confirm whether or not something malicious had occurred (such as new files in System32), but it really is only useful for technical users.

Sandboxie and DefenseWall focused on protecting particular applications or sessions, while others fell into the more traditional role of a host intrusion prevention system (HIPS), protecting critical system areas regardless of the attack vector. I was impressed with Sandboxie's ability to ensure that additional child sessions were always launched in protected mode when instantiated by a protected parent process. This is important as the browser is becoming more of a launching point for the rest of our integrated applications. Malware writers are increasingly attacking the applications as operating systems and browsers get more secure.

Nobody's perfect
Another important question is, how good is the emulation coverage? Sandbox protection products, by their very nature, don't emulate the entire operating system, as a full virtualization product such as VMware Workstation, Microsoft Virtual PC, or Parallels would. Malware programs are known to infect more than a hundred different Windows attributes, including registry locations, files, folders, startup areas, and more. How many Windows attributes and APIs are covered in the sandbox? The answer is never all. Does the product protect against remote and local buffer overflows, phishing attacks, alternative data stream techniques, file sharing avenues, and so on? Some did, most didn't.

Some of the products provided additional anti-buffer overflow, privacy, or phishing controls. The privacy and phishing controls are often already provided by other installed anti-malware programs, so their inclusion in this class of products may not be necessary (although additional layers of defense-in-depth never hurt).

Each product offered up differing levels of buffer overflow protection. For example, Sandboxie only prevented local buffer overflows if they happened against a protected process. Prevx protected the whole system against both local and remote buffer overflows, but only when they affected a critical system area being monitored.

Most of these products would not detect previously installed malware (Prevx being the exception) unless the malware made additional system modifications to the monitored areas after the products were installed. None of the products provided anti-DoS services, misconfiguration detection, missing patch analysis, or a host of other protections required to make a host system more fully secure.

Every product in this review worked only with Microsoft Windows. Some required Windows XP SP2 or later, although most worked with Windows 2000 and later versions. DefenseWall refused to defend Windows system processes. All worked with Internet Explorer and Firefox, although some of them would work with any program.

All of the products worked by installing one or more monitoring executables and services. Each provided a main executable and a system tray icon. Some of the tray icons changed colors, like a traffic light, to indicate current status (green for everything's OK to red for malware detected). All products displayed an on-screen warning when maliciousness was detected and most created log files. Interfaces ranged from Prevx's all-user elegance to Sandboxie's technical-user sophistication. The install, interface, and alerting for all products was acceptable. Pricing was $29.95 per copy or less.

Only Prevx had any enterprise capabilities, and even that was minimal. Most of the products were obviously intended for home or personal use. You won't find enterprise-wide reporting, logging, or alerting; or the capability to push out or monitor large-scale deployments. Sandbox defenses are first-generation products, sitting where anti-virus scanners were a decade ago.

Overall, this class of protection products does provide additional defense capabilities that could protect a user against unknown threats. In no case was using the vendor's product worthless, although some need to mature a bit to be ready for widespread use. The biggest question is if the additional protection value is worth the additional outlay of money and ongoing support. A fully patched system (OS and applications) where the user cannot install random programs would probably provide as much protection. How well your organization handles those two requirements will determine if sandbox products are worth investigating.

And the winners are…
In the end, the reviewer's favorite products were Prevx and Sandboxie. Prevx provided the best identification of malware and prevented most of the exploits thrown at it, though by no means all. It's nice to be told what was trying to infect your system instead of having to make trust decisions on the fly. Plus, Prevx was the only product able to detect previously installed malware, and its interface was elegant. Sandboxie was a surprise. It provided fairly accurate infection prevention and, in most cases, excellent cleanup. It requires a bit more technical knowledge when picking which changes should and shouldn't be kept, but it's free price tag makes it a winner.

Now on to the individual reviews…

Prevx 2.0
I've been a big fan of Prevx for years. It was one of the first players in the Web security space and tends to be on the cutting edge of browser defense. The product's maturity shows in the end-user interface, operational aspects, and availability in 64-bit and business versions.

Prevx provides a multipronged defense, with heavy reliance on heuristic host-intrusion detection techniques. It provides distinct protection to Internet browsers, e-mail programs, critical file and memory areas, and startup program areas, and it supplies additional defenses against keyloggers, buffer overflow programs, and network connecting malware. Although real-time monitoring and heuristics are certainly its sweet spot, Prevx contains multiple signature-based mechanisms and relies heavily on its community-based malware reporting database, which requires an active Internet connection to utilize.

[ View the companion video, "Prevx gets the slip." Download the QuickTime version. ]

After the initial licensed-based install, Prevx did a scan of some of the critical system components and checked for program updates. Prevx has the best user interface in this roundup review (see Figure 1). There wasn't a part of it I did not like. It looks good, displays what the user needs when required to make a decision, and hides when it is not in play. There are three operation modes: ABC, which is the default for beginners, and two expert modes.

When the user surfs to a malicious Web site, Prevx notes any system modifications it detects while the related files are identified and compared to a local database or sent to the larger community-based database. If identified as malware (see Figure 2), the malicious programs and system modifications are removed and the system rebooted. Suspicious programs are placed in "jail" (see Figure 3), where the user can restore or tell Prevx to quarantine or delete. Cleaning always results in a mandatory reboot, followed by an additional rescanning of critical areas and an uploading of any found changes to the community database. I especially liked this feature because it can find modifications missed on the first pass. Nice touch.

Sadly, Prevx didn't always keep my system clean. On just the fifth malware Web site, a password-stealing Trojan was able to infect the test system. Prevx had noted system changes and uploaded multiple files to the community database, but it completely missed one of the Trojan files, even after the reboot and second scan (see the program called SSUUDL in Figure 4, Figure 5, and Figure 6). In further testing on the same site, Prevx removed every infectious file nearly all of the time, but not every time. And although it detected and prevented the XP Antivirus malware program, it did not stop the Adobe Flash clipboard hijack. If Prevx could improve its accuracy, it would easily be the best product in this review.

SafeCentral 4.0
Although Authentium's SafeCentral attempts to prevent keyloggers, screenscraping software, and malware from silently exploiting systems from Firefox browser sessions, it is most proud of its ability to prevent DNS and Web site spoofing for its 15,000 registered partner Web sites. The SafeCentral Portal site list includes thousands of commonly used banking, financial, and other popular Web sites and will prevent many phishing attacks. This is an opt-in feature, forcing the user to access sites from the SafeCentral Portal in order to ensure site authenticity. If your Web site is not listed or if you are socially engineered into visiting a bogus Web site without going through the portal, you will not get the protection of SafeCentral's redirection.

[ View the companion video, "SafeCentral vs. the Adobe Flash clipboard hijack." Download the QuickTime version. ]

After you install SafeCentral, which requires a multistep process more complicated than its competitors, it loads a custom version of Firefox and modifies the toolbar in Internet Explorer, if it finds Internet Explorer on your system. Various "elements" are installed to secure and protect the desktop from the custom version of the Firefox browser and vice-versa.

When the user is in a secure Firefox browser session, the rest of the system is dimmed (see Figure 1) and interaction is restricted in significant ways. If you click any program or desktop area outside of the browser, the secure browser session is paused and dimmed. Every switch between the protected browser session and the desktop took an extra click and often caused slightly uncomfortable latency. It reminded me of Microsoft Windows Vista's "secure desktop" feature that accompanies User Account Control (UAC) protection, except that Microsoft's secure desktop provides significantly more separation and security.

In extensive testing, SafeCentral did not allow a single silent install in Firefox, except for the Adobe Flash clipboard hijack, which every other product missed as well. That's about the only good point I could give this product, and one that would be matched by a fully patched browser as well. In my testing, SafeCentral permitted hundreds of malware downloads, if the site "fooled" the user into downloading and running the program. At no time did SafeCentral stop any malware download initiated by the user, or prevent the subsequent system modification, or ever warn the user of the impending potential damage. For example, Figure 2 shows a fake MSN.com site requesting to install an update to the Adobe Flash Player.

Protection was worse for Internet Explorer. Even though SafeCentral modified the toolbar and offered an indication of alert messages, it allowed nearly every silent malware install I threw at it, without so much as a peep. Clicking the SafeCentral toolbar icon (with Internet Explorer) simply launches the further secured version of the Firefox browser, which doesn't help when visiting the millions of Web sites that require Internet Explorer. Overall, I saw no advantage to using SafeCentral with Internet Explorer and questionable value with Firefox. The strength of this product lies with its DNS and anti-phishing protection. Those who want protection against browser threats should look elsewhere.

Sandboxie 3.28
Sandboxie is a superquick download (421K) and an easy install, supporting Windows 2000 and later Microsoft operating systems. It can be used to provide sandboxed protection (for files, disk devices, registry keys, processes, threads, driver objects, named pipes, mailbox objects, events, mutexs, semaphores, sections, and LPC ports) while running any program, including any Internet browser, command prompts, and Windows Explorer. It has a multitude of configuration options and a good interface that is directed more toward technical end-users.

Sandboxie offers many runtime and configuration choices over two main views: Programs (Figure 1) and Files and Folders (Figure 2). Both figures show Sandboxie running with active malware. At any point, the user can choose to terminate sandboxed programs and delete or restore the involved objects.

[ View the companion videos, "Sandboxie sandbox under attack," "Sandboxing programs in Sandboxie," and "Sandboxie configuration options." Download the QuickTime versions: "Attack," "Sandboxing," "Configuration." ]

Because of the number of things Sandboxie emulates, it successfully stopped almost everything I threw at it, including bots, worms, Trojans, viruses, rootkits, low-level disk editing, and malicious alternative data streams. The two exceptions were, as covered in the accompanying sidebar, two tenacious tricksters, the Adobe Flash clipboard hijack and the XP Antivirus malware program. Sandboxie didn't prevent the clipboard hijack, and it did not remove all remnants of the XP Antivirus malware program when I told it to delete everything.

Still, overall, I was more impressed with Sandboxie than I expected to be -- with three reservations. First, as comprehensive as the coverage appears, Sandboxie cannot virtualize system-level drivers, which can lead to installation and stability problems from both legitimate and malicious programs. Some of the low-level malware programs I tested caused "blue screen" errors and severe booting problems afterward. To be clear, at no time did I see a malware program installed in such a way that Sandboxie allowed it to run seamlessly outside of virtualization; however, Sandboxie allowed more browser and system crashes than most of the competitors.

Second, Sandboxie only protects one program or process at a time. When you use Sandboxie, you must choose which programs and processes to protect and when. You can create one or more virtual sandboxes, each with its own settings, but what goes into each sandbox is up to the user. Occasionally, I found myself accidentally running unprotected programs when I wasn't paying attention. Plus, it's just not possible to run every program and process virtualized all the time, for various reasons (consider remotely buffer overflowed system service, anti-virus software, tape backup software, and so on), which means they can be exploited. Other competitors in this review focused on protecting critical system areas against all threats and didn't rely on the user to choose which area to defend.

Third, all trust decisions are left up to the end-user. Sandboxie never makes a declaration of safe versus unsafe content. The nontechnical end-user usually doesn't have enough knowledge of malware to make successful trust decisions. For example, Sandboxie doesn't prevent against phishing, so if a user is sent an e-mail claiming to be a security patch from Microsoft, how many end-users would download and install the patch using an unprotected browser session? How many users might be tricked by the XP Antivirus malware program? Too many, I suspect.

Overall, I liked Sandboxie's coverage, user interface, level of protection, and wealth of configuration options, especially for the price. It's a solid utility for those who can make the right trust decisions.

SoftSphere DefenseWall HIPS 2.44
Although security is rarely a binary choice, SoftSphere's DefenseWall HIPS separates all applications into just two categories: trusted or untrusted. Applications that can be expected to interface with potentially malicious content should be placed into the untrusted category. Processes started by untrusted programs automatically inherit untrusted status. Automatically downloaded or executed content from an untrusted program is protected from execution and prevented from manipulating protected resources.

Installation of DefenseWall HIPS was simple and quick, and the accompanying help file covered the essentials. The DefenseWall HIPS comes preconfigured with 11 untrusted programs, including Internet Explorer, Firefox, QuickTime, and a few other frequently exploited programs (see Figure 1). However, Adobe's Flash Player was not automatically included on this list, and this allowed the clipboard hijack exploit demo to be successful.

[ View the companion video, "DefenseWall HIPS vs. the Adobe Flash clipboard hijack." Download the QuickTime version. ]

Untrusted programs can be launched in a trusted state by choosing the application in the Untrusted applications window and selecting the Run as Trusted button. This is a nice way to end up with both trusted and untrusted browser sessions to handle various Web sites. The untrusted browser sessions are marked in the title bar to help users distinguish between the two.

The SoftSphere program has many customizable options, including the ability to include or exclude specific Windows resources (such as files, folders, registry keys, and so on) as trusted or untrusted. You can roll back any identified resource changes on a per-resource basis, although the program does not always distinguish between legitimate and malicious changes, leaving the final trust decision to the end-user. I especially like that DefenseWall HIPS considers all programs from removable sources to be untrusted by default.

Overall, DefenseWall HIPS stopped most malicious threats from automatically executing, though at times in a disconcerting way. When I browsed to malicious Web sites with an untrusted browser, the DefenseWall HIPS added any further starting programs (in testing, these were always malicious programs) to the untrusted applications list (see Figure 2), which prevented much malicious activity. Further, it warned me (Figure 3) when a malicious program was trying to modify critical system files.

Auto-downloaded malware is saved to the disk, but in such a way that it is not a threat to your system. You can disable the program, remove it, or, if the program is legitimate, move it to the trusted applications area. If you manually save a file to the desktop, which is often the case with social engineering, DefenseWall HIPS attempts to keep it marked as untrusted, but I found instances where malicious files could escape to trusted areas.

The DefenseWall HIPS includes a Stop Attack window, which allows a user to quickly close all untrusted processes if a malicious attack is suspected. The Adobe Flash clipboard hijack exploit lived through this closing; DefenseWall HIPS did not report any events or modifications, nor did it offer to roll back any changes.

DefenseWall HIPS also had a hard time cleaning up from the XP Antivirus malware, as did many of the competing programs. Although XP Antivirus was executed from within an untrusted browser session, the malware program was able to permanently modify the system and leave remnants of itself behind, even after I instructed DefenseWall HIPS to close all untrusted processes and delete all resource changes. DefenseWall HIPS did a pretty good job in stopping most malware programs, but it wasn't perfect.

ZoneAlarm ForceField 1.1
Check Point's ZoneAlarm ForceField requires Windows XP Professional SP2 or later, and works with Internet Explorer (6.0 and later) and Firefox (2.0 and later) browsers. It prevents "silent" malware downloading and keyloggers, while providing anti-phishing services, Web site inspection, and privacy components. As shown in Figure 1, ForceField can open a protected or private browser session, the latter of which adds the ability to erase browser session markers. Once a protected browser session is opened, protection is indicated by the additional toolbar (Figure 2).

ForceField accurately prevented silent infections, although it did not prevent the Adobe Flash clipboard hijack. During the first round of testing a few weeks ago I was able to bypass ForceField by using a completely different malicious Flash buffer overflow file. During the later, and longer, second round for this review, ForceField held up without allowing a single silent infection. ForceField's user interface, although not the best in the competition, certainly held its own. Warning messages were easy to understand and presented at appropriate times.

[ View the companion videos, "ForceField clears a malicious site" and "ForceField runs away with local resources." Download the QuickTime versions: "Clears malicious" and "Resource DoS." ]

For these reasons, and the fact that I have liked both Check Point and Zone Alarm software for a long time, I wanted to award a glowing review to this product. However, it's hard to give a strong recommendation to a product that only works to prevent "silent" drive-by downloads. While this is good, a fully patched system will do the same, albeit without the same level of warnings. These days, a large portion of malware is intentionally downloaded and installed by the end-user because of incredibly realistic social engineering. This is a hard deficiency to overlook.

Second, ForceField is prone to false negatives, detecting many of my very malicious test Web sites as safe or merely suspicious (see Figure 3, for example). In my testing of hundreds of malicious links, it became almost surprising to see ForceField call a Web site definitely malicious. These two complaints alone make it hard to bestow a strong recommendation, but ForceField also caused an unintentional DoS problem, which I think reveals a serious design flaw.

Frequent exploit attempts from a single Web site (which is pretty common) caused ForceField to create and re-create numerous processes (Figure 4), leading to 100 percent CPU utilization (Figure 5). (See also the video, "ForceField runs away with local resources.") Even killing the malicious browser session would not stop the DoS, as ForceField was now out of control.

Lastly, some integrated applications, such as instant messaging, can open additional browser sessions that escape ForceField's protection. And clearing ForceField's virtualized session data often removed browser settings I had hoped to keep.

ForceField does provide additional security value, but is not nearly as strong as some of the competitors. For most users, Prevx is the best choice for Web browser protection. Technical users will find a lot to like in Sandboxie.

InfoWorld Scorecard
Value (10.0%)
User interface (15.0%)
Accuracy (60.0%)
Administration (15.0%)
Overall Score (100%)
Prevx 2.0 8.0 10.0 8.0 7.0 8.2
Authentium SafeCentral 4.0 6.0 5.0 6.0 5.0 5.7
Sandboxie 3.28 8.0 7.0 9.0 5.0 8.0
SoftSphere DefenseWall HIPS 2.44 8.0 8.0 8.0 5.0 7.6
ZoneAlarm ForceField 1.1 7.0 7.0 8.0 5.0 7.3
From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies