The importance of log management using Logstash & Elasticsearch.

1. - Logs, Logs and More Logs!
Cameron Evans Lead Infrastructure Developer cameron@toplog.io

2. ● NSCC IT grad
● Backend Dev at
● Log Management Company
● Log Shipper/Receiver
● Going to say “Logs” a lot
About

3. ● Why are logs so important?
● What can we learn from logs?
● Processing logs efficiently
○ Logstash & Elasticsearch
● Searching & Visualization
● Log analysis demo
Outline

4. What is a log?
(SysAdmin nightmares are made of these . . .)

5. Timestamp + Data = Log
Data? Data.
Data!
Timestamp
● Recognize this log?
○ No standard schema
○ Easy for computer to say, hard to read
● What does this mean?
● Is this valuable information?
● What can we do with it?

6. ● TopLog cares.
○ We understand the pain
○ Server diaries
● Troubleshooting Developers
○ “Its broken”
● Your infrastructure
○ Down-time
■ Network
■ Application
Who cares?

7. Typical Security Company, inc.
● Network captures, firewalls and netflow
Not only network logs
● Firewall, switches, TCP/IP traces, NetFlow
○ Can catch intrusions
○ But how was your infrastructure affected?
● Application level logs
○ See the effects
Security and Logs

8. Syslog, web server logs, database logs, application logs
● What do we do with them?
○ Corral
○ Analyze
● Which tools to use?
○ “Wait, why should I do this again?” — The Audience
○ BFF: Logstash & Elasticsearch
Application Logs

9. ● Boss: “It’s broken”
● Developer
○ Log into server
○ Figure out the issue (skim through logs)
○ Fix the problem (hopefully)
or
○ Just get it working again
Storytime: “Its broken . . . ”

10. ● Digging through individual servers logs
○ Time consuming
○ Missing the big picture
● Every log tells a story
● “How can we manage x servers sending x/sec
events?”
“There’s got to be a better way!”

11. ● Syslog: Popular but flawed
○ Cannot ensure:
■ Integrity
● UDP based, can lose packets
○ Authentication?
■ No validation of sender
■ Plaintext
● Alternatives:
○ Syslog-ng: TCP, synchronization
○ Nsyslog: TCP, SSL authentication
● Open-source:
○ FluentD
○ Logstash
Lots o’ Log Management Tools
(syslog compatible & active community)

12. Open source - DIY
FluentD:
● Plugin-based
● SaaS options
● CRuby
● No Windows support
● Less familiar
○ Feel free to chime in
FluentD vs Logstash
Logstash:
● SaaS options
● Plugin-based
● JRuby
○ Java Dependency
● Major OS support
● Lightweight Forwarder
● Great community
○ “If a user has a bad
time, its a bug”
○ IRC
● Elastic Family

13. ● Log-Management tool
● Event-Processing Pipeline
● Queue-based
● Plugins
○ Input (receive logs: file, lumberjack, syslog, redis)
○ Filter (parse, modify, concatenate, conditionals)
○ Output (store logs: elasticsearch, nagios, syslog)
○ Tailor to your system
● Scalable
● Parse log types into filterable/searchable fields
● Open-Source Community
Logstash Server

14. Example Logstash Input Config
● Files, stdin
● Forwarders,
○ UDP, TCP, Twitter, s3
● Queues
○ Redis, RabbitMQ, ZeroMQ

15. Example Logstash Filter Config
● How would you like your event?
● Parse, modify, concatenate, conditionals
● Custom filter plugins
● What does this do?

16. Example Logstash Filter
● Regex!

17. ● Store:
○ Elasticsearch, s3, Hadoop, Redis, Syslog
● Notify:
○ Email, Slack, Hipchat, Pagerduty, Nagios
● React:
○ exec, jira,
● Can tailor to your system
○ Simple or Complicated solutions
Example Logstash Output Config

18. ● Distributed search engine
● Document-based data store
● Open-Source
● Runs on Apache Lucene
● Super-fast full-text filtering & searching
● Visualization
● Aggregations
● Horizontally scalable
● Communicate using HTTP JSON requests
Elastic Search

19. ● Can be compiled anywhere with Go installed
○ Linux, OSX, Windows, etc
● Lightweight footprint
● Own proprietary ‘Lumberjack’ protocol
○ SSL communication channel to server
○ Encrypted & authenticated
● Handles multiple files, stdin
● Reliable & Resilient
○ No data-loss
○ Persistent connection
Logstash Forwarder

20. Example Logstash-forwarder Config

21. ● User requests the server
● Server logs the request
● Streamed by Logstash-Forwarder
● Processed by Logstash
All Together Now

22. ● Stored in Elasticsearch as
● Search, Filter, & Aggregate all in JSON
● Visualization
○ Toplog, Kibana, Marvel, Graphite
All Together Now

23. Our Infrastructure

24. We aren’t the only ones:

25. Hands on walk through
● Take a peek behind the curtain
● Please ask questions!
Two VMs:
● LEMH web server:
○ Logstash-Forwarder
● Log storage Server:
○ Logstash server
○ Elastic search

26. Visualizing and interacting
● Can’t just rely on raw HTTP requests
● Interface tools: Kibana, Marvel, Graphite
○ Visualize
○ Filter, Search & Aggregate
○ How do you know what to look for?
● Moving forward:
○ “Meta-meta-data”
○ Knowledge base of patterns & behaviours
○ Predictive Analysis

27. Let’s talk about us ba-by
● Automated pattern-extraction
● Pattern correlation
● Behaviour detection
● Anomaly detection

28. More info
● Presentation slides @ https://blog.toplog.io
● FFFound: https://www.found.
no/foundation/elasticsearch-from-the-bottom-
up/
● Logstash IRC
● Think of a question later?
cameron@toplog.io
● More on what we do?
https://toplog.io team@toplog.io