An overview of fluentd for the San Antonio DevOps Meetup

1. Hart Hoover | @hhoover
Josh O’Brien | @OBrienCommaJosh
@SADevOps

2. fluentd

3. Log all the things
to all the things!

4. …in JSON!

8. input&{&
&&file&{&
&&&&path&=>&"/var/log/http.log"&
&&}&
}&
filter&{&
&&grok&{&
&&&&match&=>&[&"message",&"%{IP:client}&%
{WORD:method}&%{URIPATHPARAM:request}&%
{NUMBER:bytes}&%{NUMBER:duration}"&]&
&&}&
}

12. Fluentd Events
Time Tag
Record
passed from source
or
adding a parsed time
message
routing in fluentd
JSON

13. #&receive&events&via&HTTP&&
<source>&&
type&http&&
port&8888&&
</source>

14. #&read&logs&from&a&file&&
<source>&&
type&tail
path&/var/log/httpd.log&
format&apache
tag&apache.access&&
</source>

15. #&DOCKER&OMG&
<source>&
&&type&tail&
&&format&json&
&&path&/var/lib/docker/containers/ID/IDYjson.log&
&&pos_file&/var/lib/docker/containers/ID/IDY
json.log.pos&
&&tag&docker.container&
&&rotate_wait&5&
&&read_from_head&true&
</source>

16. #&save&alerts&to&a&file&&
<match&alert.**>&&
type&file&&
path&/var/log/fluent/alerts&&
</match>

17. #&save&access&logs&to&MongoDB&&
<match&apache.access>&&
type&mongo&
database&apache&
collection&log&&
</match>&

18. #&Post&to&IRC&
<match&**>&
&&type&irc&
&&host&localhost&
&&port&6667&
&&channel&fluentd&
&&nick&fluentd&
&&user&fluentd&
&&real&fluentd&
&&message&notice:&%s&[%s]&%s&
&&out_keys&tag,time,message&
&&time_key&time&
&&time_format&%Y/%m/%d&%H:%M:%S&
&&tag_key&tag&
</match>

19. #&forward&other&logs&to&servers&&
<match&**>&&
type&forward&
&&<server>&&
&&&&host&192.168.0.11&&
&&&&weight&20&&
&&</server>&
&&<server>&&
&&&&host&192.168.0.12&&
&&&&weight&60&&
&&</server>
</match>

20. #&Send&logs&to&ElasticSearch&
<match&**>&
&&type&elasticsearch&
&&logstash_format&true&
&&host&localhost&
&&port&9200&
&&index_name&fluentd&
</match>

22. Client Libraries
Ruby
Python
Java
PHP
Node.JS
Scala

23. https://github.com/treasure-data/chef-td-agent
Install with Chef
Vagrantfile
& kitchen.yml included!

24. https://github.com/dmytro/fluentd-cookbook
Possibly Better
https://gist.github.com/hhoover/4fceb09148a73f45136a

25. @SADevOps
http://meetup.com/SanAntonioDevOps