To log or not to log? This is an age-old question for developers. Logging everything can be great because you have plenty of data to work from when you have a problem. But it’s not so great if you have to grep and inspect it all yourself. In my mind, developers should instead be thinking about logging the right events in the right format for the right consumer. Instrumentation is one of the most valuable sources of data you can log. At Loggly, we instrument like crazy in our code because we need to understand what our system is doing and where we can improve it. When you’re dealing with billions of events per day, the ability to quickly identify hot-spots is invaluable. However, there are lots of different opinions about what you should instrument and concerns about the performance impact of that instrumentation. What follows are the guidelines and best practices that we’ve developed to help ourselves do a better job of instrumentation. Hopefully they’ll help you too… Read the full blog post on the Loggly blog → http://bit.ly/BestLoggingPractices

1. To Log or Not to Log:
Proven Best Practices for
Instrumentation
Apache Storm
Jon Gifford
Founder and Chief Search Officer
January 2014
| Log management as a service Simplify Log Management

2. About Me
15+ years of search experience
§ Lots of different search products
§ Lots and lots of machines
§ Lots and lots of logs...
Bias:
§ Lucene, Java, *nix, Distributed, Realtime,
§ Measure, Measure, Measure
| Log management as a service Simplify Log Management

3. Loggly: Addressing the first big data
problem every company faces
§ Centralized logging
and archival
§ Real-time processing,
analysis and
visualization
§ Monitoring, alerting
and troubleshooting
| Log management as a service Simplify Log Management

4. To Log or
Not to Log?
+ - yourself
Lots of
data to solve
operational
problems faster!
Incredibly
painful to
grep and
inspect
| Log management as a service Simplify Log Management

5. Instrumentation:
Your Most Valuable Log Data
For us, instrumentation is the act of gathering
performance data about your application and
surfacing it via logs.
§ Provides a low-impact mechanism for gathering
production system performance data
§ Visualization is key to understanding this data
| Log management as a service Simplify Log Management

6. Best Practices for
Instrumentation
| Log management as a service Simplify Log Management

7. Best Practice #1
Treat instrumentation as an ongoing,
iterative process.
§ Start by logging everything at a high level
§ Instrument deeper every time you ask, “What
happened here?”
§ Instrument deeper in areas that are more variable
or expensive than you thought
| Log management as a service Simplify Log Management

8. Best Practice #2
Always instrument anything that goes
out of process.
§ Can be indicative of larger
problems, like the canary in the
coal mine
§ Example: Instrument calls to
your database, even though
they should always be fast
§ Distributed system problems are
not well behaved – by definition
| Log management as a service Simplify Log Management

9. Best Practice #3
Always log unacceptable performance.
§ Log “soft exceptions” (e.g. slow queries)
§ “Trust but verify”
| Log management as a service Simplify Log Management

10. Best Practice #4
If possible, log enough context so you
can get the complete picture of what
happened from a single log event.
Elapsed time
Process
Log this
… not this!
End
Start End
| Log management as a service Simplify Log Management

11. Best Practice #5
View machines, not people, as your
logs’ consumer.
| Log management as a service Simplify Log Management

12. Best Practice #6
Trends tell the story better than data points.
§ Graph performance
over time to see
daily fluctuations
§ Use rollups where
it’s not practical to
log every single
event
| Log management as a service Simplify Log Management

13. Best Practice #7
Instrumentation is NOT a substitute
for profiling, and vice versa.
Instrumentation Profiling
Your production environment is not the same as development!
| Log management as a service Simplify Log Management

14. Best Practice #8
Flying slower is better than flying blind.
§ “Speed advantage” is
illusory if you can’t
understand system
performance
§ Our debate: not
whether to instrument
but how much
| Log management as a service Simplify Log Management
Photo by Eric Kilby

15. Implement Instrumentation
Best Practices Faster and
Better with Loggly.
Try Loggly for Free! → http://bit.ly/BestLoggingPractices
About Us:
Loggly is the world’s most popular cloud-based log management solution, used by
more than 5,000 happy customers to effortlessly spot problems in real-time, easily
pinpoint root causes and resolve issues faster to ensure application success.
Visit us at loggly.com or follow @loggly on Twitter.
| Log management as a service Simplify Log Management

16. Proven and Trusted by 5,000+ your peers…
“If we didn’t have
Loggly, there would
be days where our
revenue would be
affected by as much
as 70 percent.”
– Albert Ho,
Executive Producer/
Platform Product Manager,
Rumble Entertainment
| Log management as a service Simplify Log Management