Until recently, sending logs to Graylog without using Syslog or any third party program was a bit cumbersome. This has changed since version 1.1. Graylog now has its own log collector which is tightly integrated with the Graylog server and web interface to simplify the management of log shippers. The Graylog collector runs on several operating systems including Linux, Windows, Mac OS and AIX. It makes it easy to send data like Apache access logs or Windows event logs to Graylog without the need of any third party tools. In this talk I will introduce the Graylog collector and show how to install and configure it on Linux and Windows. I will also show how to extract structured data from those logs and an example integration with the Icinga monitoring system to alert on critical events.

1. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Monitoring Linux and Windows Logs
with Graylog Collector
Bernd Ahlers
Graylog, Inc.

2. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Structured Logging & Introduction to
Graylog Collector
Bernd Ahlers
Graylog, Inc.

3. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Introduction: Graylog
● Open source log management platform
● Collect, index and analyze structured and
unstructured log data
● Alerts based on log data
● Extensible via custom plugins

4. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

5. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

6. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

7. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

8. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

9. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

10. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

11. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

12. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
More about Graylog
● www.graylog.org
● marketplace.graylog.org
● docs.graylog.org
● github.com/Graylog2

13. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Why are we writing logs?
● Getting insight & collecting business metrics
● Debugging problems
● Building an audit trail
● Monitoring

14. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
How do we access our logs?
● Applications write to local files
● SSH into machines
● tail, grep, awk
● If lucky: central log management

15. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
What do they look like?
● Syslog RFC 3164 (BSD)
● Syslog RFC 5424

16. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Syslog RFC 3164 (BSD)
Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD
(command -v debian-sa1 > /dev/null && debian-sa1
1 1)

17. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Syslog RFC 5424
2003-10-11T22:14:15.003Z mymachine.example.com
evntslog - ID47 [exampleSDID@32473 iut="3"
eventSource="Application" eventID="1011"] BOMAn
application event log entry...

18. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Apache
127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100]
"PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910
"-" "Mozilla/5.0 (Linux) mirall/1.7.1"

19. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Postfix
Aug 5 17:05:26 hostname postfix/qmgr[308]:
A44F828C71: from=<bamm@example.com>, size=153136,
nrcpt=1 (queue active)

20. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Squid
sq18.wikimedia.org 1715898 2010-12-
01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200
13208 GET
http://en.wikipedia.org/wiki/Main_Page NONE/-
text/html - - Mozilla/4.0%20(compatible;%20MSIE
%206.0;%20Windows%20NT%205.1;%20.NET%20CLR
%201.1.4322) en-US -

21. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
log4j
0 [main] INFO MyApp - Entering application.
36 [main] DEBUG com.foo.Bar - Did it again!
51 [main] INFO MyApp - Exiting application.

22. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Ruby Logger
I, [2015-11-18T00:16:27.723972 #3609] INFO -- :
Hello world!

23. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
#1 Problem: Timestamps
● Everyone likes to invent one
● Missing most of the time: timezone, year

24. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
How to get value out of unstructured logs?
● Regex
● More regex
● Even more regex

25. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:
[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|
1dd|[1-9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4})
{1,2})|:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-
9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:
[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-
4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]
{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]d|1dd|[1-
9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]
{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-
5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d))
{3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-
Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|
1dd|[1-9]?d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]
{1,4}){0,5}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|
1dd|[1-9]?d)){3}))|:)))(%.+)?

26. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Grok
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9...
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
HOSTNAME b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-
Za-z-]{0,62}))*(.?|b)
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
...
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

27. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Graylog: Extractors
● Regular expressions based
● Extracts data into message fields

28. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

29. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
How to fix this?
● Central log collection (Graylog, ELK, others)
● Use structured log formats
– Structured Syslog RFC 5424
– CEF Format
– GELF
– JSON

30. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Structured Syslog RFC 5424
2003-10-11T22:14:15.003Z mymachine.example.com
evntslog - ID47 [exampleSDID@32473 iut="3"
eventSource="Application" eventID="1011"] BOMAn
application event log entry...

31. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
CEF by ArcSight/HP
Sep 19 08:26:10 host CEF:0|HP|siem|
1.0|100|service
successfully stopped|10|
src=10.0.0.1 dst=2.1.2.2 spt=1232

32. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
GELF
{ "version": "1.1",
"timestamp": 1385053862.3072,
"host": "example.org",
"short_message": "A short message",
"full_message": "Backtrace herennmore stuff",
"level": 1,
"_user_id": 9001,
"_some_info": "foo",
"_some_env_var": "bar"}

33. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
JSON
{ "source": "example.org",
"message": "A log message",
"timestamp": "2015-11-15T10:43:21Z",
"user_id": 9001,
"http_method": "GET"}

34. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
How we try to improve the ecosystem
● Icinga2 GELF output for events
● Docker GELF logging driver (since Docker 1.8)
● apache-mod_log_gelf (beta)
● log4j2-gelf
● gelfclient Java library
● svloggelfd (log forwarding for runit)

35. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
We at Graylog <3 structured data
and you should too!

36. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Introduction: Graylog Collector
● Reads local log files and ships them to Graylog
● Windows EventLog support (limited for now)
● Transport encryption via TLS
● Runs on Linux, Windows, Mac OS X and AIX

37. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Why another Collector?
● There are lots of others: nxlog, fluentd, heka,
filebeat, rsyslog, syslog-ng
● We want integration and centralized
management of collectors in Graylog

38. Bernd Ahlers – Graylog, Inc. bernd@graylog.com

39. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Collector Installation
● OS packages for Linux distributions
● Manual installation on Windows via ZIP file
(MSI upcoming)
Runs as Windows service

40. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Collector Configuration
server-url = "http://your-graylog-server:12900"
inputs {
windows-application-log {
type = "windows-eventlog"
source-name = "Application"
}
}
outputs {
gelf-tcp {
type = "gelf"
host = "your-graylog-server"
port = 12201
}
}

41. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Collector: Current State
● Windows EventLog support needs update to
support new Windows APIs
● File reading needs improvement
● Centralized management needs to be
implemented
● :-(

42. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Tomorrow: Hackathon

43. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
Thank you!
Thank you for your time!

44. Bernd Ahlers – Graylog, Inc. bernd@graylog.com
QA
Ask me anything!
Bernd Ahlers / Graylog, Inc.
bernd@graylog.com
@berndahlers
www.graylog.org
github.com/Graylog2