Learn to love your log files

How to stop worrying and turn your big, dumb pile of impenetrable event data into instant actionable intelligence

Follow @rogeragrimes

Considering how much information is available in log files, you'd think companies would pay more attention to them. Client computers, servers, firewalls, network devices, and other appliances generate reams of event logs every day, but these logs often go ignored.

Although it's a security sin, it's understandable on many levels. First, logs can contain vast amounts of uninteresting events. In fact, most logs are nothing but noise. With the rare exception, most logs are close to useless. At one current client, 1,000 computers and one perimeter firewall generate 25GB of log files on a daily basis. Out of that, in a typical week, not a single event is a true security issue requiring an immediate response. Oh, security events do happen, but when they do, they are normally buried in a sea of unimportant noise.

[ Honeypots are a sweet solution to the insider threat. Learn how to secure your systems with InfoWorld's Security Adviser newsletter. ]

Second, log file review is rarely a management priority -- until a tipping point event occurs or the auditors complain loud enough. Third, when the staff is already overworked, messing with something that provides so little real-time value seems wasteful. Lastly, few people get excited about reviewing log files. The answer to "Hey, Johnny, what do you want to be when you grow up?" is never Log File Reviewer.

So why care about log files? Because most malicious exploits and intrusions leave their fingerprints all over the log files. If the log file management system was crafted correctly, it could provide true real-time value. In this column, I'll attempt to give those of you hoping to improve your log file management system the CliffsNotes version of how to pull off a successful program.

Sawing logs
You can start by reading NIST's Special Publication 800-92, "Guide to Computer Security Log Management." Released in September 2006, it's unusually easy to read for a NIST publication and extremely useful for deploying event log management systems in the real world. It's considered the gospel for this small corner of the computer security world.