InfoWorld review: Better network security, compliance with log management
ArcSight, LogRhythm, and NitroSecurity ace mining event logs for security alerting, compliance auditing, and other usesFollow @rogeragrimes
Log collection. Collecting the log information from all the various monitored clients is the backbone of any log management product. Most products have both agentless methods and client agents to collect logs. Not having an agent means administrators don't need to distribute, install, and configure additional software to every client. However, agentless log collection still requires planning and work. Most products collect logs using syslog forwards, WMI queries, or other remote methods (the last two usually require client administrative passwords). All require the necessary rule modifications if firewalls are involved. Whatever you do, don't think of agentless as no work or it will surprise you.
Client agents have benefits that agentless collection methods have a hard time meeting. Most agents have multiple configuration options that allow administrators to have finer-grained control over what events are collected and how. For instance, instead of sending every log message to the centralized server, an agent can just send critical events, and store the rest locally for later retrieval if needed. Client agents can often offer transmission compression, allowing more events to be sent in less time and with smaller network bandwidth utilization, although it's doubtful you'll get the superior compression statistics that each vendor advertises in real-life scenarios.
Monitored clients can be added one at a time (usually via IP address or domain name), using mass importing (to add multiple devices at once), or using some sort of initiated querying process (usually through Active Directory browsing or IP address scanning). Most products allowed "device groups" to be created, to collect one or more monitored clients under a given group name as determined by some attribute -- for example, by device type, IP address, or name. Device groups can then be monitored as a single entity making alerting and reports easier to accomplish when trying to focus on a particular device class.
Client agents can also be used to store events, in case the centralized log management tool is offline. One of the best features of the most sophisticated agents is in measuring network and/or local CPU utilization and throttling back the message send rate until the congestion clears up. Lastly, many agents have a "heartbeat" feature that will send warnings if the client has not transmitted messages in a certain time period, although this can be mimicked identically with "zero baseline" alerts as well server-side only. Not surprisingly, ArcSight, a longtime SIEM leader, has more client agents than any other competitor.
As covered above, the more parsed data a log management product has, the faster and more efficient the product can be when sifting through large amounts of data looking for a particular data interest. A big differentiator of products is how many parsers the product comes defined with. Some of the leaders, like ArcSight, are bundled with well over 100 data collectors defined. On the lower end, some products only have a few dozen parsers or will claim that their generic parsers are identical in efficiency. But in general, the closer the parsers mimic your environment, the better (but don't let this be the sole decision point). Some log management products allow administrators to create their own parsers, which could prove very useful in many environments.
One relevant additional note: Most products claim to have Windows event log collection agents. However, many of these agents were made prior to Microsoft's latest Windows versions and don't have a good understanding and parsing of the more granular logs and views in these later operating systems. Many of the parsers and agents understand the three conventional default logs -- Application, Security, and System -- but cannot allow the administrator to choose from among the 100-plus built-in views that Windows Vista, Windows 7, and Windows Server 2008 provide.
Splunk is one tool that understands the new Windows log formats. However, I didn't find a tool that works easily with the newer Windows' built-in, event forwarding technologies (even if the product was hosted on a Windows OS and easily could use the newer technologies). Windows' own event forwarding could be used in place of all the other agent and agentless methods. As with most product categories, log management hasn't kept up with the latest client changes.