Icinga Camp Stockholm 2019 - Monitoring Logs with Icinga and Elastic Stack

2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm
Icinga 2
and Logs

• Thomas Widhalm ( @widhalmt )
• Lead Support Engineer @ Netways
• Specialised in Icinga and Elastic Stack
• Collector of Star Wars Lego and
Camo patterns
About me

About me

Logs

• Every IT infrastructure has lots of them
• Many admins don‘t really care
• Focused view of one box
• Filters and parsing on the fly
– Different levels of knowledge
– Rerun all filters every time you change something
• Full harddisks or now long time storage
• Only used when something bad happened
Logs

Different kinds of logs

• Logs of monitored objects
• Logs of Icinga
• Logs of alerts and notifications
• Logs of logmanagement
Logs and Icinga

• Give more thorough insight
• Allow monitoring of otherwise inaccessible objects
• Different ways of monitoring
– Plugin / Agent
– Logmanagement
Logs of monitored objects

• Show how Icinga is doing
• Help with monitoring and debugging
• Very useful for post mortems and support tickets
Logs of Icinga

• Part of Icinga logs
• Can be used for SLA / umbrella monitoring
• Basically show how monitored objects are doing
Logs of alerts and notifications

• Often can‘t be processed by logmanagement itself (Loops!)
• Show problems in logmanagement infrastructure
Logs of logmanagement

Toolset

• No native way of monitoring logs
• Plugins for monitoring logs
– Statusmonitoring, no searching
– Only single hosts
Icinga

• Collects Logdata from many sources
• Stores data in a central database
• Monitoring addon
– Not free
– Cumbersome configuration
• Connectors to many receivers
– Cumbersome configuration
– All but flexible
Elastic Stack

• Search Server (based on Apache Lucene)
• Elastic, highly available, load balanced very resilient
• Extremely scalable
• REST-API for communication
Components of Elastic Stack: Elasticsearch

• Receive Logs from many sources
– Syslog
– Beats (Agents)
• Send to many targets
– Elasticsearch
– Icinga
• Parse, disect, transform, filter, enrich Logs
Components of Elastic Stack: Logstash

• Webinterface for Elastic Stack
• Search and filter logs
• Build Dashboards for Screens or interactive drill drown
Components of Elastic Stack: Kibana

• Lightweight agents
• Collect Filelogs (syslog) or Event Log
• More specialised beats available
– Icingabeat
– MySQL-beat
– Redisbeat
Components of Elastic Stack: Beats

Common problems

• Hardly any „problem event“ has a corresponding „ok again“ event
• Hearing nothing from your hosts:
– Everything is fine
– Too dead to talk
End of the world or end of the problem?

• Automatically return to „OK“ after a while
– Send notifications
– Enrich with active monitoring
• Have someone check
– Can create lots of work
Best effort

Approaches

• Use Elastic Stack to collect and store Logmessages
• Use Icinga for alerting
• Different ways of connecting
Combined forces

• Cumbersome configuration
– Still easier than full blown logmanagement?
• Not part of monitoring plugins
• Independent from everything else
– Use to avoid loops
– Monitor logmanagement infrastructure
Ye goode olde check_logfiles

• Full blown logmanagement solution
• Several ways of ingestion
– Reads logs from filesystem
– Receives logs from „icingabeat“ agent
• Several ways of monitoring
– „icinga“ output to API
– Icinga Web module „elasticsearch“
Elastic Stack & Icinga

$ yum install java-1.8.0-openjdk-devel
$ /usr/share/logstash/bin/logstash-plugin install
logstash-output-icinga

output {
icinga {
host => 'fornax.icinga-book.local'
user => 'root'
password => '***'
action => 'process-check-result'
action_config => {
exit_status => 0
plugin_output => "%{[message]}"
}
icinga_host => '%{[host]}'
icinga_service => 'logevent'
}
}

• Use all actions of the API
– Process check result
– Add hosts
– Set downtimes
• Decide which data to use from what logevent in Logstash config
• Get results into Icinga in almost no time
• Use passive checks with automatic recovery

Icinga Web Modules „elasticsearch“

• Configure connection to Elasticsearch
• Set filters to identify logs
– Objectname in Icinga = Objectname in logs!
– Enforce Icinga Web permissions on logs
• Give Icinga users quick access to logs without allowing access to
Kibana

# icingacli elasticsearch check --instance elastic01-hot --crit 5 --warn 3
--index logstash* --filter "beat.hostname=qa,source=/var/log/httpd/*.log"
--from -5m
OK - 0 hits
Elastic Stack and Icinga

• Ready-to-use ruleset
– https://github.com/Icinga/icinga-logstash-pipeline
• Ingest Icinga Logs
• Have data parsed from logs
– „eventtype“ for every type of event
– Data like endpointnames, objectnames etc extracted
– Numbers like queuelengths extracted

• Use as a standalone Logstash pipeline
– Input and output for Redis are provided
– Clone git repo into configuration directory (and use *conf files)
– Rest of minimal config is provided in Readme
• Collect logs from masters, satellites, agents
• Get the whole picture what‘s going on in your monitoring

Filter for severity

• Restrict to facility or severity
• Watch for spikes in logs
• View message just like in the logfile

View details of an event

• Get Facility/Severity
• „Eventtype“ for every kind of logmessage
• Get related object (split into host, service, notification etc.)
• Get message specific details (pluginoutput, exitcodes etc.)
• Use all these fields for filters or graphs

Use dashboards

• Use dashboards for screenwalls
– See problems arising before they get critical
– Get fresh status update during problems
• Use them interactively
– Klick on parts of graphs to create quick filters
– Make fast drilldowns
– Filter every item on the dashboard at once (graphs, event lists)

Detect anomalies

• See unusual spikes in event flows
– Big benefit even for experienced log-greppers
• Get consolidated logs from all (or some) nodes
– Drill down to the problem at hand

Questions and Answers

• thomas.widhalm@netways.de
• thomas.widhalm@icinga.com
• Twitter: @widhalmt
• GnuPG: B50D AF2B 22A6 94E8 C195 9C89 DAAC 19AE A84C B603
Contact

netways.de
blog.netways.de
git.netways.de
sales@netways.de
netways
netways
netways
+49 911 92885 - 66
Contact

Icinga Camp Stockholm 2019 - Monitoring Logs with Icinga and Elastic Stack

Recomendados

Recomendados

Más contenido relacionado

Más de Icinga

Más de Icinga (20)

Último

Último (20)

Icinga Camp Stockholm 2019 - Monitoring Logs with Icinga and Elastic Stack