InfoWorld review: Better network security, compliance with log management
ArcSight, LogRhythm, and NitroSecurity ace mining event logs for security alerting, compliance auditing, and other usesFollow @rogeragrimes
Log storage. Storing tens of millions to billions of messages takes a lot of disk space. Most appliances come with terabytes of disk storage in RAID configurations. Both software and hardware products claim to do some sort of storage compression, but as with transmission compression claims, take the vendors' figures with a grain of salt. Their storage compression statistics are often based upon the smallest event log messages and the highest compression values, and don't reflect real-world results.
Still, it's important to find out from the vendor whether the product is software or appliance, what is the maximum disk space (or file size) the product supports, and in what configurations. What RAID arrays are supported? Different RAID configurations have different performance characteristics -- that is, some are faster at writing and some are faster at reading -- so flexibility is a plus. Does the vendor support digital signing of collected log data for attestation needs?
Most products have a maximum log size as well, having to do with limitations of the underlying host OS. If the product is an appliance, can the data be stored to external drive arrays? How much data can actively be indexed and easily retrievable? Every product allows data to be exported or archived. Exported data typically is kept offline and must be imported en masse to be searchable. A few solutions handle this more flexibly. For example, LogRhythm allows administrators to define a filter to import only the needed data instead of everything.
A few products have what is known as "storage groups," which are individually defined logical partitions devoted to a particular task, such as PCI compliance, or a particular grouping of devices -- for example, Cisco wireless routers. In addition to organizing a certain class of data for reporting purposes, storage groups can be used to make sure that a particular application has enough disk space to serve a particular policy requirement -- for instance, save data for two years. ArcSight is especially strong in this area, with sizing parameters and CPU prioritization available.
Lastly, you'll want to determine whether event log data is stored or archived in the vendor's proprietary format, in its raw (unfiltered and unstructured) form, or in both? Most products store active data in a proprietary format, but archived or exported data remains in a raw format. This means that re-imported data will have to be parsed and indexed again to be useful, but it's also easier to prove chain-of-custody concerns if that raw data (assuming it is also digitally signed) is later needed for legal reasons.
Real-time viewing. Most products allow real-time viewing of incoming data and show some top trends -- often called tailing. If you have a system of any moderate size, with hundreds to thousands of messages coming in every second, real-time viewing of all data quickly loses its allure. All products allow real-time data to be filtered to show only relevant events for a particular interest. Often these filters can be saved to search historical data and produce related reports.
The best real-time viewers allow users to click on specific data fields to pivot to new views. For instance, maybe you're viewing incoming data about a particular workstation and you see a suspicious TCP port. In some products, clicking on the port value could switch the current real-time view to show all workstations using the same port. Other products can only do this on historical data or require that you switch views into an "investigator" mode. All of the products tested provide pretty flexible viewing, though LogLogic and LogRhythm were strongest.