Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)

8.614 visualizaciones

Publicado el

Dennis shows us how to integrate the ELK (ElasticSearch, Logstash & Kibana) stack with Zabbix.

Publicado en: Tecnología

Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)

  1. 1. Xifeo ICT | Dennis Kanbier Monitoring the ELK stack using Zabbix (and Grafana) Dennis Kanbier 26 Nov 2015 - NLZGG
  2. 2. Xifeo ICT | Dennis Kanbier Overview • Introduction • What is ELK and why do I want it? • How to monitor ELK with Zabbix • Impress people with pretty graphs • Demo time ( please tweet using #nlzgg! )
  3. 3. Xifeo ICT | Dennis Kanbier Who am I? • Dennis Kanbier • dennis.kanbier@xifeo.nl • Linux Consultant with Xifeo ICT • “dkanbier” on Zabbix forums and IRC
  4. 4. Xifeo ICT | Dennis Kanbier What is the ELK stack • Elasticsearch • Search server • Based on Apache Lucene • Logstash • Data pipeline • Processes logs and other data • Plugins • Kibana • Web frontend for Elasticsearch
  5. 5. Xifeo ICT | Dennis Kanbier Basic Logstash Pipeline
  6. 6. Xifeo ICT | Dennis Kanbier Apache log example • 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/ logstash-monitorama-2013/images/ahiruyaki.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36” • A few interesting fields: • client_ip: 83.149.9.216 • status code: 200 • timestamp: 04/Jan/2015:05:13:42 +0000 • and much more
  7. 7. Xifeo ICT | Dennis Kanbier { "clientip" : "83.149.9.216", "ident" : , "auth" : , "timestamp" : "04/Jan/2015:05:13:42 +0000", "verb" : "GET", "request" : “…/a-2013/images/ahiruyaki.png”, "httpversion" : "HTTP/1.1", "response" : "200", "bytes" : "203023", "referrer" : “http://semicomplete.com …“, "agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X…” } Logstash processes the log message and produces output:
  8. 8. Xifeo ICT | Dennis Kanbier Now what? • Use the processed data to do fun things like: • plot all entries using the [timestamp] field to see how busy the web server is during the day • which part of your website gets requested the most using the [request] field • monitor the status of your web server using the [response] field • Or do even more fun things using plugins…
  9. 9. Xifeo ICT | Dennis Kanbier logstash-filter-geoip • Use the geo-ip filter plugin while logstash is processing the message • Looks up the geographical location of an IP address and adds this information to the message • Use Kibana to create a heat map based on the geographical information in each message to see from which part in the world your website gets visited • Very nice to have in case of DDOS attacks originating from places you don’t expect visitors from • Or to determine where to up your advertising game
  10. 10. Xifeo ICT | Dennis Kanbier Example heat map
  11. 11. Xifeo ICT | Dennis Kanbier Developer issues
  12. 12. Xifeo ICT | Dennis Kanbier Not always the developers fault
  13. 13. Xifeo ICT | Dennis Kanbier Why developers love ELK • Logs from all components and servers are available in one central place, easily searchable using the Kibana web interface • No more begging the scary Unix administrator for log files • Log message are enriched with meaningful fields which enables users to: • query on stack traces • query on specific thread id’s • query on log_level • query on customer_id • list faults per server/application/cluster/log_type/etc.. • You can easily plot logs, see how fast requests are handled • Timestamps are normalised, even log entries from different time zones are searchable chronologically
  14. 14. Xifeo ICT | Dennis Kanbier Example Kibana screen
  15. 15. Xifeo ICT | Dennis Kanbier Monitoring Logstash with Zabbix • Used 3 components to generate data and send it to the Zabbix server: • logstash-input-heartbeat • logstash-filter-metrics • logstash-output-zabbix
  16. 16. Xifeo ICT | Dennis Kanbier logstash-input-heartbeat • Plugin to generate a heartbeat message at a configurable interval • The message contains a field called “clock” with the epoch time of the moment it was send • Assign a type of “heartbeat” to the message so we can use that to filter out the message further in the pipeline and send it to the Zabbix server
  17. 17. Xifeo ICT | Dennis Kanbier logstash.conf input { heartbeat { message => "epoch" interval => 20 type => "heartbeat" } } filter { } output { }
  18. 18. Xifeo ICT | Dennis Kanbier logstash-output-zabbix • Used to send data to Zabbix • Requires a configured Zabbix host and trapper item to receive the data • These fields must be added to the [@metadata] field of the message you want to send to the Zabbix server
  19. 19. Xifeo ICT | Dennis Kanbier logstash.conf input { heartbeat { message => "epoch" interval => 20 type => "heartbeat" } } filter { if [type] == "heartbeat" { mutate { add_field => { "[@metadata][zabbix_key]" => "logstash_heartbeat" } add_field => { "[@metadata][zabbix_host]" => "logstash-indexer-1" } } } } output { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => clock }
  20. 20. Xifeo ICT | Dennis Kanbier Results in Zabbix • Zabbix host logstash-indexer-1 should already have the item with key “logstash_heartbeat” configured as a Zabbix trapper item. • Logstash now sends the epoch value every 20 seconds to the logstash_heartbeat trapper item • You can use this to further process the data in Zabbix: • trigger when there is too much time between epoch values • trigger when you do not receive any values for a period of time
  21. 21. Xifeo ICT | Dennis Kanbier Example item in Zabbix
  22. 22. Xifeo ICT | Dennis Kanbier logstash-filter-metrics • Plugin to count messages and flush the result on a specific interval • Used to monitor logstash performance by counting messages and store their count value in Zabbix as Delta (speed per second) • which gives us the messages this logstash server handles per second
  23. 23. Xifeo ICT | Dennis Kanbier logstash.conf input { … } filter { metrics { meter => "events" add_tag => "metric" add_field => { "[@metadata][zabbix_key]" => "logstash_events" } add_field => { "[@metadata][zabbix_host]" => “logstash-indexer-1“ } flush_interval => 20 } } output { if "metric" in [tags] { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => “[events][count]” } } }
  24. 24. Xifeo ICT | Dennis Kanbier More results! • We now have the item “logstash_events” which shows how many messages we process per second:
  25. 25. Xifeo ICT | Dennis Kanbier Now about that last slide…. • Presenting reports and graphs are not exactly Zabbix’ strongest points • Works all right to view single values like on the previous slide • But try to align multiple graphs nicely on one screen to put on a wall and impress people
  26. 26. Xifeo ICT | Dennis Kanbier Grafana • Build to present metric data in a simple and nice manner • Supports multiple data sources like Graphite and InfluxdB • Alexander Zobnin created grafana-zabbix, which enables us to use the Zabbix database as a Grafana backend through the Zabbix API • https://github.com/alexanderzobnin/grafana-zabbix
  27. 27. Xifeo ICT | Dennis Kanbier Example dashboard
  28. 28. Xifeo ICT | Dennis Kanbier Things to consider • Grafana queries the Zabbix database through the API • If you have a lot of graphs with a lot of items these queries are quite expensive • Especially if you have multiple dashboards refreshing automatically on a high interval • Still learning on how to best counter these challenges
  29. 29. Xifeo ICT | Dennis Kanbier Live demo time! Did you remember to tweet things with hashtag #nlzgg?
  30. 30. Xifeo ICT | Dennis Kanbier Questions?
  31. 31. Xifeo ICT | Dennis Kanbier Contact details • dkanbier on Freenode IRC • https://github.com/dkanbier • http://denniskanbier.nl/blog for tutorials • dennis.kanbier@xifeo.nl Resources • http://www.zabbix.com • https://github.com/alexanderzobnin/grafana-zabbix • https://www.elastic.co • http://www.xifeo.nl

×