This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana).
Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015.
More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report
7. 7 CONFIDENTIAL
Default problem
• Multiple log time formats
•Apr 28 20:21:59
•[27/Apr/2015:07:05:28 +0000]
•071012 09:27:32
•Mon, 27-Apr-15 06:27:02 UTC
•2015-04-28 20:07:51 +0000
• Starts not with timestamp or without timestamp
•Error messages with really unhelpful info
•No rotation
•No scaling
Tools?
• grep
• awk / sed / cut
•less / tail
•vi / vim
•regular expression
•...
8. 8 CONFIDENTIAL
Logging Solutions
Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial
Logstash Logstash shipper or
logstash-forwarder
(Lumberjack)
RPM installation
Logstash shipper or logstash-
forwarder (Lumberjack, encrypted
transport is the default)
RPM installation
Output plugins
Central server-master with a hot-
standby in case of failure
Codecs plugins
Grok debugger
ElasticSearch,
MongoDB,
AWS S3
and much more
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
fluentd Input plugins
Install from source
or via gem
Output plugins
Load-balance between multiple
hosts or have a master with a hot-
standby in case of failure
Plugins Doesn’t provide any
storage tier itself but
allows you to easily
configure where your logs
should be collected
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
splunk Splunk Universal
Forwarder
RPM installation
Splunk Universal Forwarder
SSL security
Splunk Splunk Splunk Splunk Splunk YES
Graylog2 Graylog2
Logstash
Graylog2
Logstash
Graylog2 ElasticSearch Kibana,
graylog2
graylog2 graylog2 NO
loggly loggly loggly loggly Hosted loggly loggly loggly YES
9. 9 CONFIDENTIAL
What is ELK, and why do we need it?
• ELK is a stack of programs that help dealing with logs.
• Includes:
– Aggregation of logs
– Search capabilities
– Aggregation of statistics
– Visualizations
12. 12 CONFIDENTIAL
Filters➔ grep
➔ date
➔ json
➔ grok
➔ ....
http://logstash.net/docs/1.4.2/ - Full list
Why Do I like Logstash?
It uses Grok filter for parsing standard and non standard logs:
Log Line:
27/10/14 07:39:28 [localhost-startStop-1] [] INFO
com.vidmind.config.LoggingPropertyPlaceholderConfigurer -
streams.limit.general = 0
Pattern:
%{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}