Considering how much valuable information is available in log files, you'd think more companies would pay attention to them. Workstations, servers, firewalls, appliances, and other computer devices generate reams of event logs every day, and despite mountains of evidence showing their practical, cost-saving uses, logs often go ignored. A good log management system can help significantly with security, application troubleshooting, compliance, and systems management. If that's the case -- and it is -- why do logs and log management sometimes still get a bad rap?
It's understandable on many levels. First, logs can contain towering amounts of uninteresting, hard-to-decipher events, burying more useful information. In fact, without the appropriate tools and filters, logs can be nothing but noise -- and lots of it.
[ Get the full scoop on getting more value from your log files in the InfoWorld "Log Analysis Deep Dive" PDF special report. | Better manage your company's information overload with our Enterprise Data Explosion newsletter. ]
A standard Microsoft Windows computer can easily generate thousands of events each day even when things are humming along without a real problem. A thousand computers can generate tens of gigabytes of log files on a daily basis. I've seen enterprise event log collector tools bring robust networks to their knees. What's worse, many administrators would tell you that in a typical week, not a single issue requiring an immediate response was uncovered. "Talk about a waste of resources," they will tell you, even as valuable, useful data is passing under their eyes.
Diamonds in the rough
Log file review is rarely a management priority -- until it hits a tipping point or the auditors complain loud enough. Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed. Yet management tends to act as if logs aren't worth the time and effort to analyze, a dismissal that trickles down to overworked staff. Why mess with something that seems like a waste of time to all parties involved?
Another factor is simply human nature: Few people get excited about reviewing log files. The answer to "Hey, Johnny, what do you want to be when you grow up?" is never log file reviewer, even if a good log reviewer is actually worth his or her weight in gold.
So why should you or your company care about log files? Because they allow an IT organization to be proactive versus reactive. The typical IT department waits for calls for help before responding to problems. But by the time end-users call in, they are already frustrated, the event that prompted the call has typically entered a critical phase, and IT is forced to respond in the most inefficient manner possible.