Like Google for log files
Splunk brings enterprise search techniques to log file analysis
As much as one system log file can tell you, correlating data from multiple logs can tell you a lot more -- whether you're troubleshooting a Web application, looking for network bottlenecks, or chasing down a network security incident. A new product from startup Splunk Technology, aptly named Splunk, takes a lightweight, Web-search-style approach to all-purpose log analysis.
Splunk builds a searchable index of machine data from multiple log files, organizes the data into events, and allows you to search through these events by time, text string, and keyword via Web browser. You can combine search terms using Boolean operators, and you can refine searches just by clicking values in the data -- user names, domain names, IP addresses, transaction IDs, and more.
In addition to seeing distinct event types in the data, Splunk's algorithms determine degrees of similarity and difference among those events. Here's where the real magic is: Use the keyword "related" to find events that contain the same values, "similar" to find events that contain values that are nearly the same, and "unexpected" to find those troublesome outliers -- the events that don't conform to the historical pattern. In a flash, Splunk will show you a ranked list of related, similar, or unexpected events from all of your log sources.
Splunk Team Server is due by year's end. Splunk Enterprise Server will follow in the first quarter of 2006. In addition to supporting multiple hosts and role-based access for multiple users, these editions will have monitoring and alerting capabilities, tunable indexing, and import and processing features to support distributed environments. Meanwhile, you can get an idea of what Splunk can do by downloading the free Splunk Personal Server at splunk.com.
Splunk Personal Server
Cost: Free download
Available: Public beta available now; Version 1.0 due August