1. Graylog2 use cases for
distributed web applications
Manage your logs in the dark and have lasers
going and make it look like you're from space
Lennart Koopmann, 2010
www.lennartkoopmann.net / www.graylog2.org

2. It's a DevOps thing.
Compose meaningful and structured log
messages to allow easy analysis and searching.

3. Bad:
- Could not repair image foo.jpg
- Could not repair image bar.jpg
- Could not repair image baz.jpg – Invalid header checksum.
- Missing POST param 'creditcardnumber'
- Payment of John Doe did not succeeed.

4. Good:
- [runner][repair-broken-images]Could not repair image
foo.jpg – File not found.
- [runner][repair-broken-images] Could not repair image
bar.jpg – File not found.
- [runner][repair-broken-images] Could not repair image
baz.jpg – Invalid header checksum.
- [payment][checkout] Missing POST param 'creditcartnumber'
CUSTOMER #1337
- [payment][backend] Payment of CUSTOMER #1337 did not
succeeed.

5. Which images were broken?
repair-broken-images.+repair images(.+.jpg)s.s(.+)
foo.jpg
File not found.
bar.jpg
File not found.
baz.jpg
Invalid header checksum.

6. Why did the payment fail in the
backend?
payment].+CUSTOMER #1337
[payment][checkout] Missing POST param 'creditcartnumber'
CUSTOMER #1337
[payment][backend] Payment of CUSTOMER #1337 did not
succeeed.

7. Message type distribution
payment
runner
payment-backend
payment-checkout
runner-image

8. Define log guidelines
Just like your usual coding guidelines.
(slap everybody who does not follow them with a large trout )

9. Use case 0:
The usual stuff.
Use Graylog2 to monitor your applications from the inside. Analyze
your logs, see if something goes wrong, receive warnings when
messages rates climb over a given level. Check the logs regularly
to identify problems.

10. Use case 1:
Developer logs.
Use GELF and give every developer his own hostname like
yourapp-johndoe – Now create a stream for every developer. Voilá:
No more tail -f debug.log and Graylog2 sugar from the beginning of
your development cycle.

11. Use case 2:
Important messages
Imagine you do some kind of domain registration for customers.
This stuff likes to fail and you want to be informed when it does and
why it did. Create a stream that fetches all failed domain
registrations and subscribe to it by email (released in v. 0.9.4) to be
notified instantly.

12. Use case 3:
Streams of certain application parts.
You have some scripts searching for broken images, deleting or
repairing them that are running the whole day. Create a stream
that fetches all messages from a runner and get a live output of
what it is doing right now. You could also create a blacklist instead
of a stream if you don't want to bug others with the messages. Get
warnings like in use case 2 when something goes wrong.

13. Use case 4:
Live tail at release.
You are releasing a new version of your application today. Start the
live tail (released in v. 0.9.4) to see what is happening in your
system in real time.

14. Use case 5:
Activity log.
A user blames the support that you deleted all his content. How to
debug this? Would be not such a big problem if you had logged
every activity of your users to Graylog2. Blacklist [activitylog]
and Log messages like [activitylog] USER #45262 DELETED
image25526. Search for what you need with blacklist disabled.
(released in v. 0.9.4)

15. Important:
Use structured and meaningful messages.
Have logging guidelines. (and follow them)
Choose severity with care: You might be called in the night once
that EMERG message arrives.
Don't log useless messages. That will be the clutter that ruins your
analysis, statistics and warning levels.
Already think of what to log in your problem analysis steps.