This document summarizes a StatsD workshop presented by Daniel Schauenberg at Monitorama 2013. The workshop covered what StatsD is, the different types of metrics it can handle, and how to set up and configure StatsD and its backends like Graphite. It also provided examples of sending metrics to StatsD and ideas for extending and customizing StatsD functionality.

1. StatsD workshop @
Monitorama 2013
Daniel Schauenberg
Software Engineer, Etsy Infrastructure
@mrtazz
Friday, March 29, 13

2. • What is this StatsD?
• Metrics types
• Backends
• Graphite and operating StatsD
• Send all the metrics
• Write all the backends
Friday, March 29, 13

3. Friday, March 29, 13

4. Metrics
processing
module
Receiver
loop
Backends
Friday, March 29, 13

5. legacyNamespace: false
Friday, March 29, 13

6. Counters
foo.bar:1|c
stats.counters.foo.bar.rate
stats.counters.foo.bar.count
Friday, March 29, 13

7. Gauges
foo.gar:30|g
foo.gar:-5|g
foo.gar:+5|g
stats.gauges.foo.gar
Friday, March 29, 13

8. Sets
foo.sar:30|s
foo.sar:50|s
stats.sets.foo.sar
Friday, March 29, 13

9. Timers
foo.tar:30|ms
stats.timers.foo.tar.mean_90
stats.timers.foo.tar.std
stats.timers.foo.tar.sum_90
Friday, March 29, 13

10. Histograms
histogram: [{metric: "foo.tar", bins:
[10,20,30,50]}]
stats.timers.foo.tar.histogram.bin_10
stats.timers.foo.tar.histogram.bin_20
Friday, March 29, 13

11. Friday, March 29, 13

12. Pluggable backends
function MyBackend(startupTime, config, emitter){
var self = this;
this.lastFlush = startupTime;
this.lastException = startupTime;
this.config = config.mybackendconfig || {};
// what to do with metrics
emitter.on('flush', function(timestamp, metrics) {});
// update your status
emitter.on('status', function(callback) {});
// handle raw packet
emitter.on('packet', function(packet, rinfo) {});
};
Friday, March 29, 13

13. Management Interface
> telnet 127.0.0.1 8126
> stats
> counters
> delcounters
> help
Friday, March 29, 13

14. Graphite + StatsD
• default backend
• whisper archives should match flush
interval
• sum aggregation for raw counters
• docs/graphite.md
Friday, March 29, 13

15. The single-thread
dilemma
Friday, March 29, 13

16. Tuning UDP
# increase default core memory sizes
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.ipv4.udp_wmem_min = 67108864
net.ipv4.udp_rmem_min = 67108864
net.ipv4.udp_mem = 4648512 6198016 9297024
Friday, March 29, 13

17. Sampling
• default sampling of 0.1 on all metrics
• most sent keys (log file) are sampled 0.01
Friday, March 29, 13

18. Quis monitoret ipsos
monitores?
Friday, March 29, 13

19. Friday, March 29, 13

20. Friday, March 29, 13

21. Friday, March 29, 13

22. Friday, March 29, 13

23. Let’s do this
Friday, March 29, 13

24. Basic setup
• git clone https://github.com/etsy/statsd.git
• curl https://gist.github.com/mrtazz/5270376
> test_config.js
• nc -k -l 2003 > output.log
• node stats.js test_config.js
Friday, March 29, 13

25. Send some stats
> echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125
> echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125
> echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125
> echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125
> echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125
Friday, March 29, 13

26. Some actual stats
Disk space:
df -m | grep "/var" | awk '{print "diskspace.var:"$4"|g"}'
Apache HTTPD processes:
echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g"
Logfile size:
du -s /var/log/chef.log | awk '{print "cheflog:"$1"|g"}'
Friday, March 29, 13

27. Is this real life?
• Set up a web page
• Have it send stats to StatsD
• page load time
• counters when page was accessed
• look in the examples/ directory for code
Friday, March 29, 13

28. Histograms
• You should have timers now
• Configure histograms for them
• { graphite: { histograms: [] }
Friday, March 29, 13

29. Level up!
Friday, March 29, 13

30. Write a new backend
• Have it HTTP POST to a web server
• write metrics to a file
• send a txt message
• repeat all UDP packets to TCP
Friday, March 29, 13

31. Hackathon ideas
• Metrics path validation in backend
• Pluggable receiver frontends
• Forked backends
• Configurable timer metrics
Friday, March 29, 13

32. • #statsd on Freenode IRC
• statsd@librelist.com
• Ask me
Friday, March 29, 13

33. Basic setup
• git clone https://github.com/etsy/statsd.git
• curl https://gist.github.com/mrtazz/5270376
> test_config.js
• nc -k -l 2003 > output.log
• node stats.js test_config.js
Friday, March 29, 13

34. StatsD workshop @
Monitorama 2013
Daniel Schauenberg
Software Engineer, Etsy Infrastructure
@mrtazz
Friday, March 29, 13