How we do logging at Databox with help from logstash-forwarder, Logstash and Kibana,...

1. Logging @ Databox
Oto Brglez / @otobrglez

2. What is logging?
/../ Logging, or commercial logging,
involves cutting trees for sale as timber or
pulp. The timber is used to build homes,
furniture, etc and the pulp is used to make
paper and paper products. Logging is
generally categorized into two categories:
selective and clear-cutting.
Selective logging is selective because
loggers choose only wood that is highly
valued, such as mahogany.
Clear-cutting is not selective. Loggers are
interested in all types of wood and
therefore cut all of the trees down, thus
clearing the forest, hence the name- clear-
cutting. /../

3. Logs
● System logs (syslog, syslog-ng,..
● Services (nginx, mysql, cassandra,... )
● Applications
○ Errors (php-fpm /var/log/php-fpm/… )
○ “Events” (/var/log/my_app.log)

4. Architecture @ Databox
new
… more servers
...
*.log logstash
forwarder
<-->
lumberjack
protocol log.databox.
com
logstash elastic
search

5. Logstash Forwarder

6. Logstash
● input (lumberjack,
redis, *MQ,..)
● filter (gork)
● output (elasticsearch,
redis, …)

7. ElasticSearch
● Search server
● RESTful
● JSON documents
● Lucene based
● Highly scalable
● Java based
● Apache License w/
Vendor

8. Kibana
/../ Kibana is an open
source data visualization
platform that allows you
to interact with your data
through stunning, powerful
graphics that can be
combined into custom
dashboards that help you
share insights from your
data far and wide. /../

9. Demo
log.databox.com

10. Tips
● Log more!!!
○ Log important steps
○ Log exceptions
○ Log errors
○ Log events
○ Log *
● Make sure crashes are recorded
somewhere and that we all know
about it!

11. Thanks!
● Questions?
@otobrglez