elaticsearch

1. ELK Stack- Elasticsearch, Logstash and
Kibana

2. So what’s a log

3. Log = timestamp + data

4. Lifecycle of a log
Real
Delet
e

5. Default problem
• Multiple log time formats
•Apr 28 20:21:59
•[27/Apr/2015:07:05:28 +0000]
•071012 09:27:32
•Mon, 27-Apr-15 06:27:02 UTC
•2015-04-28 20:07:51 +0000
• Starts not with timestamp or without timestamp
•Error messages with really unhelpful info
•No rotation
•No scaling
Tools?
• grep
• awk / sed / cut
•less / tail
•vi / vim
•regular expression
•...

6. Logging Solutions
Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial
Logstash Logstash shipper or
logstash-forwarder
(Lumberjack)
RPM installation
Logstash shipper or logstash-
forwarder (Lumberjack, encrypted
transport is the default)
RPM installation
Output plugins
Central server-master with a hot-
standby in case of failure
Codecs plugins
Grok debugger
ElasticSearch,
MongoDB,
AWS S3
and much more
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
fluentd Input plugins
Install from source
or via gem
Output plugins
Load-balance between multiple
hosts or have a master with a hot-
standby in case of failure
Plugins Doesn’t provide any
storage tier itself but
allows you to easily
configure where your logs
should be collected
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
splunk Splunk Universal
Forwarder
RPM installation
Splunk Universal Forwarder
SSL security
Splunk Splunk Splunk Splunk Splunk YES
Graylog2 Graylog2
Logstash
Graylog2
Logstash
Graylog2 ElasticSearch Kibana,
graylog2
graylog2 graylog2 NO
loggly loggly loggly loggly Hosted loggly loggly loggly YES

7. What is ELK, and why do we need it?
• ELK is a stack of programs that help dealing with logs.
• Includes:
– Aggregation of logs
– Search capabilities
– Aggregation of statistics
– Visualizations