Look deep into log files
Omnisight offers powerful, flexible analysis if you have the skillsFollow @pvenezia
Addamark provides a few sample files, but it would be great to see more included with the package. Omnisight is very powerful and flexible, but it’s also very complex. You can use it to analyze log files from any application, in any format, whether supported by Addamark or not, but it requires significant skill to do so. Most of the customization of the import tools is done in a mixture of Perl and SQL. To correctly implement Omnisight will require a high level of Perl and SQL experience, although Addamark does include seven days of assistance in the cost.
Once a log file has been imported it can then be analyzed by the engine. Querying and reporting is done via a CLI and Web front-end, both of which are minimalist interfaces. The Web interface provides a central interface for reporting, query construction, and maintenance, while the CLI tools are broken into separate functions.
Reports rely on queries to the database, and SQL queries must be written for the specific log file to be analyzed. Addamark provides a handful of queries that highlight the sample log files, but again, significant skill is required to write queries for anything beyond that. Once queries have been written, they can be collected into reports to be run by an administrator, or by an authorized user. No reports are provided with Omnisight; all reports must be developed in-house.
With its format-agnostic approach, Omnisight can be adapted to handle just about any log file analysis task or objective. An obvious use is to collect and analyze data from a variety of network devices to investigate a suspected employee or an external break-in. Another adaptation could be cross-analyzing log files generated by security-card access devices and the PBX to determine if anyone who wasn’t logged entering the building was making phone calls.
Addamark is aiming Omnisight at large infrastructures with heavy-duty log-file storage, maintenance, and analysis requirements, hence the built-in clustering. Log files imported into Omnisight are meant to stay there ad infinitum, not discarded after a period of time. To achieve this, every log file is compressed during import. On a five-node cluster of dual Xeon servers, a 500MB Check Point firewall log in LEA (Log Export API) format was imported, mirrored, and compressed in 125 seconds, with a nearly 10:1 compression ratio. On the same cluster, a fairly complex query of 6 million records returned in 23 seconds — truly impressive feats.
Omnisight is a powerful and flexible tool for log-file analysis. The scope of its reach is almost limitless, given its open architecture and highly customizable parsing functions. This is not a tool to be installed by administrators and driven by nontechnical management; it’s a tool to be carefully implemented and maintained by skilled programmers. If you have the need to store and analyze massive log files from a wide variety of services and devices — and if you have the skill to handle the implementation — Omnisight can handle the load.