Security landscape shifts as technologies combine

Protecting your network environment demands new strategies as technologies merge and mature

In today's era of perimeter-invading worms, malicious e-mails that don't rely on attachments, and tenacious spyware, safeguarding the enterprise demands a security framework that marshals a more sophisticated combination of technologies. A traditional firewall and an up-to-date virus scanner may no longer be enough.

But what is enough, exactly? Getting a handle on which solutions to deploy -- and where -- has become increasingly difficult, as emerging technologies have begun to overlap and functionalities have merged.

For example, if your new firewall can block application-layer threats, do you need an intrusion detection system? Should you choose a rules-based IDS, or one that uses anomaly detection to flag zero-day attacks? And when should you consider host-based security measures, or a specialized application security solution?

Naturally, the answers to these questions depend largely on the value of the assets you're trying to protect. In any case, it's critical to keep an eye on the changing landscape of point solutions. By keeping abreast of security advances, you'll be better positioned to capitalize them before newly evolving threats infiltrate your enterprise.

Firewalls and IDSes

Firewall vendors such as Check Point Software Technologies and Juniper Netscreen are touting new application-layer filtering capabilities, and these are important advances. After all, if your firewall is intelligent enough to block a DoS attack or a NetBus Trojan probe, you can rest so much easier.

Nevertheless, compared to a well-tuned IDS, even the most modern firewall is a blunt instrument -- and necessarily so. A stateful inspection firewall is an effective way to block unauthorized port traffic, defend against IP address spoofing, and thwart other, more recent types of attacks. Proxy firewalls, which prevent direct connections to hosts inside the network, provide yet another layer of protection.

But all firewalls have holes, if only because they must remain open to legitimate traffic. They can't inspect the contents of point-to-point VPN traffic, and even those that do make application-layer decisions can identify only a narrow range of threats that ride almost universally welcome protocols such as UDP and HTTP.

More and more malicious attackers are using port 80, which is almost always open between segments. In fact, if I were a malicious coder, I'd look first to port 80 -- or another commonly opened firewall port -- in order to gain entry to a network. To counter this, you need the data-level inspection that only an IDS or IPS can provide.

Detect or Prevent?

Because they can prevent malicious exploits, IPSes are outpacing IDSes as the preferred security systems of choice. After all, if an IPS can prevent an attack, why would you ever choose an IDS instead?

The problem is that many, if not most, IDSes and IPSes suffer from high percentages of false positives. And, whereas an IDS will only log a false positive, an IPS will block traffic marked as potentially dangerous, thereby preventing a significant amount of legitimate traffic from entering your network. Although vendors are working on improving accuracy, accidentally denying legitimate traffic can be even more catastrophic to your business than failing to block a malicious attack.

IDSes and IPSes are the best solutions for preventing buffer overflows -- second to patching, that is -- and for recognizing abnormally constructed data. IDSes and IPSes excel at inspecting packet data and lower-level packet information, which is what makes them so effective at identifying threats. But whereas IPSes will block identified threats, IDSes simply alert administrators after identifying malicious traffic.

IDSes and IPSes are located either at a network filtering point -- to identify threats passing between networks -- or on a host computer. Host-based IDSes and IPSes are designed to protect only the host on which they are located. While network-based IDSes and IPSes will identify -- and in the case of an IPS, stop -- general threats, host-based solutions are configured to protect systems against malicious attacks targeting specific operating systems or application software. For example, an IPS for Microsoft SQL Server will be designed to prevent SQL injections and guesses at database passwords.

IDSes and IPSes use two types of technology, the most commonly employed of which is fingerprinting. Fingerprinting, aka pattern-detection, solutions work much the same way anti-virus scanners do, that is, making use of databases that store predefined malicious byte patterns to identify specific threats. Perhaps because this approach is the most popular, fingerprinting databases must be constantly updated; they can be defeated by new and slightly modified threats.

The second technology, anomaly detection, uses baseline profiling to recognize statistically deviant traffic patterns. For example, an anomaly-detection solution would flag high levels of sustained network traffic originating from a low-traffic host, or it would notice unauthorized manipulation of system files. Anomaly-detection engines are useful for detecting zero-day or slightly modified exploits.

Which technology should you choose in an IDS or IPS? Ideally, as InfoWorld's reviewers discovered in a recent product roundup, the solution should contain both components. Most threats have an easily recognizable byte pattern, but anomaly detection should be layered on top of this pattern-detection capability in order to discover threats that do not have a specific signature and to stop zero-day exploits.

Early adopters placed IDSes and IPSes outside the firewall or on the DMZ to complement external security defenses. Unfortunately, reported events quickly overwhelmed administrators. Today, IDSes are deployed inside the trusted network as an early warning system to notify administrators when the perimeter has been compromised. Host-based IPSes, such as Sana's Primary Response, may be the most effective way to lock down specific Web and database servers.


Anti-virus technology isn't a panacea. Traditional anti-virus scanning solutions that use pattern databases have been great at detecting already-known threats but have proved terrible at dealing with zero-day exploits, slightly modified one-off malicious programs, and buffer overflows. The SQL Slammer worm infected tens of thousands of computers in less than 10 minutes. If anti-virus scanners can't defeat zero-day attacks, how can they be expected to deal with zero-minute attacks?

Vendors have responded by incorporating heuristic scanning tools and by submitting more frequent database updates. Similar to anomaly-detection technology, heuristic scanners analyze files by looking for coding actions often related to malware, such as modified executables, self-contained SMTP engines, and writing to sensitive registry areas. Heuristic technology isn't new, but vendors are increasing their efforts to make it more accurate by minimizing false positives. Vendors have also recoded anti-virus programs to check for and download pattern databases more frequently. Whereas updating databases weekly used to be often enough, today's anti-virus tools need to be checked daily or at least have the updates pushed to them as soon as a new threat is identified. Unfortunately, anti-virus scanners will never be 100 percent accurate, and all it takes is one unpatched system or one computer lacking an up-to-date anti-virus scanner to infect the whole enterprise.

As a result, vendors are developing ways to quarantine infected computers and those that don't meet corporate security policy. Several anti-virus vendors offer solutions that will cut off network traffic to and from computers that don't meet predefined criteria. Trend Micro's Network VirusWall appliance will check computers for patch status, enforce the use of up-to-date anti-virus software, and isolate infected machines.

Because malware can infect a computer from dozens of different vector -- the Internet, removable media, and p-to-p channels -- the best location for anti-virus software is on the desktop, given that. But no matter how malware arrives, it must execute on the desktop to infect the computer. By placing defenses on the desktop, you can detect malware regardless of how it arrives. Defending on e-mail servers is another good strategy, because most worms and viruses arrive via e-mail -- although this trend won't last forever. And many entities are placing anti-virus solutions on gateway devices in order to inspect network traffic. Although in theory the gateway would be an opportune position from which to catch malware, scanning each network packet against a large signature database will significantly slow down network throughput. In practice, most gateway solutions scan only a few popular protocols such as SMTP, HTTP, and FTP. That leaves a whole lot of other ports and protocols for malware to exploit.

Network Quarantining

This year's hottest security technology is network quarantining. No matter how strong your network security defenses are, if one misconfigured computer connects to your network, then it's game over -- a lesson that was driven home by Slammer and Blaster.

Network quarantining solutions prevent computers that are not properly configured, not patched, or not running updated anti-virus software from connecting to the network. New arrivals are pushed onto a restricted network and inspected. If the computer meets security policy, it is approved and is allowed to connect to the regular network.

Dozens of vendors are developing network-quarantine solutions, including Check Point, Cisco, McAfee, Microsoft, Trend Micro, and Zone Labs. Some solutions such as Microsoft's Network Access Quarantine Control require specific server and client software. More frequently, quarantining requires special network appliances or software that interfaces with existing routers or switches to handle blocking on the network layer or below. Unfortunately, quarantining is not easy to implement, and the bugs are still being worked out.

Battling Spam

Anti-spam filters are becoming increasingly sophisticated, with accuracy rates in the high 90s being the norm. The best solutions combine Bayesian filtering and content inspection. Most use some combination of Bayesian filtering and content analysis along with whitelists and blacklists.

As a general rule, accuracy improves the farther away you get from the desktop. In test after test, desktop solutions such as those from McAfee, Microsoft, and Symantec fare the poorest. ASP solutions such as MessageLabs and Postini are among the most accurate. InfoWorld has also found BrightMail, MiraPoint, and Proofpoint to be very good at blocking spam and avoiding false positives.

Many anti-spam solutions also contain anti-virus mechanisms. Some perform simple file-attachment blocking, and others contain anti-virus scanning functionality. File-attachment blocking is easy to beat, so products using proven anti-virus solutions fare better in removing legitimate threats. As long as you pick an accurate product, anti-spam solutions don't have too many disadvantages beyond the initial expense and setup. The biggest worry is that false positives might block legitimate e-mails, but with training and adjustable scoring, false positives can be minimized.

Security solutions are improving as major vendors combine various technologies into single-offering packages. Unfortunately, software is becoming more complex, and malicious attackers are getting bolder. Until the root causes of computer exploits -- poor programming practices and lack of persuasive authentication -- get resolved, your enterprise will need multiple defenses.

At a minimum, that means a late-model firewall, network anti-virus, an anti-spam gateway, and e-mail filtering. Midsize and larger companies should also deploy an IDS/IPS and consider a network-quarantine solution.

Even the best solutions are undermined by poor user practices and untrained administrators. Spend as much time on these issues as you do investing in technology.