Moving towards unified logging covers thoughts on moving from proprietary log consolidation tools to open source options such as Elasticsearch, Logstash, Kibana (the ELK stack) along with other ideas like using FluentD in place of Logstash.

1. Phil Cryer, September 2014
Moving towards unified logging

2. goal
=> decouple data sources from backend systems by providing a
unified logging layer to route logs as needed

3. currently
=> A host runs a Splunk app and forwards all of its logs to Splunk

4. Host
a host

5. Splunk
Host
sends logs to splunk
Splunk

6. => this works, but we want a more flexible, open source solution
that doesn’t restrict us with specific tools or size quotas
why is this a problem

7. => create a unified logging layer to handle logs with FluentD, an
open source, flexible and lightweight alternative to route logs
idea
http://www.fluentd.org/

8. Host
the ELK stack in development…

9. Elasticsearch
Host
Logstash
logstash writes to elasticsearch

10. Elasticsearch
Host Host
Logstashrsyslogd
but this can be done just with rsyslogd

11. Elasticsearch
Host Host
Logstash FluentD
and can also be done with FluentD
Host
rsyslogd

12. Elasticsearch
Host Host Host Host
FluentD
Host
Logstash FluentD Dockerrsyslogd
FluentD
but FluentD can be used for more, like routing

13. Elasticsearch
Host Host Host
FluentD
Host
Logstash FluentD Docker
FluentD
Host
rsyslogd
and handle input from various data sources

14. Elasticsearch
Host Host Host
FluentD
Host
Splunk
Logstash FluentD rsyslogd
FluentD FluentD
Host
rsyslogd
and output to various backends, even Splunk

15. Elasticsearch
Host Host Host
FluentD
Host
Splunk
Host
Logstash FluentD rsyslogd Splunk
FluentD
Host
rsyslogd
it could do this independently of Splunk

16. Elasticsearch
Host Host Host
FluentD
Host
Splunk
Host
Logstash FluentD rsyslogd Splunk
FluentD FluentD
Host
rsyslogd
or in parallel

17. Elasticsearch
DB Memcache DNS
FluentD
Host
Splunk
IDS
Logstash FluentD rsyslogd
FluentD
App
rsyslogd
and these could be from a variety applications
rsyslogd
FluentD

18. Docker TCP Socket
FluentD
Host
MySQL Scala App
Data sources
AWS
using various data sources
http://www.fluentd.org/datasources
AWS, Docker Containers, Flume, Java Apps, MySQL Slow
Query Logs, Scala Apps, TCP Socket, and more
DB Memcache DNS IDSApp

19. FluentD
Host
Kafka
FluentD
HDFS
FluentD
Data outputs
sent to various data outputs
http://www.fluentd.org/dataoutputs
AWS, Kafka, CouchDB, Elasticsearch, Hbase, HDFS, Mongo DB,
MySQL, Redis, Riak, Splunk, and more

20. Moving towards unified logging
Thanks.
so let’s start…