One of the big challenges of distributed systems is managing lots of log files. You could spend time collating log files from each and every microservice instance, every container, and every VM in your cloud infrastructure, or you could implement a unified logging layer to collect, collate, and store that data for later analysis.
Fluentd is a commonly used data logging layer with a large and growing community of developers, as well as support from key cloud service providers, including Amazon and Microsoft. For Microsoft it’s a key component of the cross-platform Operations Management Suite, and it’s supported by the newly open-sourced PowerShell.
Once in place in your environment, Fluentd uses plugins and support for common logging platforms to extract data, filter it, and route the results to an appropriate storage or analysis tool. Data is restructured in JSON format, so it can be processed by endpoint APIs.
The result is a fast and powerful tool that can help with day-to-day operations, as well as providing formatted data that is used by more specialized systems for monitoring or security analysis. One big user is Line, which drops log data directly into Hadoop clusters to do real-time analysis of the messaging service that’s currently accessed by some 600 million users.
-- Simon Bisson