This document discusses how to measure everything by collecting a million metrics per second with minimal developer overhead. It describes using the StatsD, Graphite, and CollectD tools to collect and visualize metrics from applications, services and infrastructure. Specific techniques are provided for instrumenting applications, Apache, Varnish, and other systems to capture metrics and send them to StatsD and Graphite for storage and visualization.
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
How to Measure Everything: A Million Metrics Per Second with Minimal Developer Overhead - PuppetCo
1. HOWTO MEASURE EVERYTHING
A million metrics per second with minimal developer overhead
!
Jos Boumans - @jiboumans
http://www.imagemediapartners.com/Portals/20286/images/MeasuringTape-s.jpg
7. AVERAGE DATA EVENTS / SEC
http://investor.fb.com/results.cfm
http://www.statisticbrain.com/twitter-statistics/
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm
0 35,000 70,000 105,000 140,000
Twitter: NewTweets Wikipedia: PageViews
Facebook: Messages Sent Krux: New Data Points
11. METRICS &VISUALIZATION
… and a little bit of monitoring
http://getfit101.files.wordpress.com/2012/04/visualization.jpg
12. VISUALIZATION MATTERS
Humans are good at patterns & shapes
http://1.bp.blogspot.com/-CO-8FK9bohE/T89rD8dTyEI/AAAAAAAAAEE/YUZ00v_filk/s1600/live_like_it_matters_by_mythirll-d3iqcxt.jpg
13. INSIGHT MATTERS
We consider it a core competence
http://yourselfseries.com/teens/files/2013/05/suicide_bonus_Insight_final.jpg
14. SHOW EVERYONE
And better yet, encourage people to add their own
http://www.kissimmee.org/ftp/KCC/events/views/images/crowd_cheer.jpg
24. INFRASTRUCTURE
Everything needed to create, capture and
act on a million metrics per seconds
http://discussamerica.org/remer-blog/images/Freeway_Interchange2.jpg
27. STATSD
Simple stats collector service
https://github.com/etsy/statsd
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.phphttp://emps.exeter.ac.uk/media/universityofexeter/emps/eisa/exista-splash.jpg
28. STATSD NAMING SCHEME
stats. # to distinguish from events
$environment. # prod, dev, etc
$cluster_name. # api-ash, www-dub, etc
$application. # webapp, login, etc
$metric_name_here. # any key the app wants
$hostname # node the stat came from
33. STANDARD AGGREGATIONS
# Average & Sum for timers
<prefix>.timers.<key>._totals.ash.<type>.avg (10) =
avg <<prefix>>.timers.<<key>>.<node>.<type>
!
<prefix>.timers.<key>._totals.ash.<type>.sum (10) =
sum <<prefix>>.timers.<<key>>.<node>.(?!upper|lower)<type>
!
# Min / Max for Lower / Upper
<prefix>.timers.<key>._totals.ash.upper (10) =
max <<prefix>>.timers.<<key>>.<node>.upper
!
<prefix>.timers.<key>._totals.ash.lower (10) =
min <<prefix>>.timers.<<key>>.<node>.lower
http://graphite.readthedocs.org/en/latest/config-carbon.html#aggregation-rules-conf
35. GRAPHITE ALTERNATIVES
Circonus:All the insights you ever wanted
Zabbix: OSS self hosted monitoring http://circonus.com
http://zabbix.com
https://github.com/lyft/circonus-statsd-backend
https://github.com/dlecocq/statsd-zabbix
36. GRAPHITE.JS
Custom dashboards using jQuery
https://github.com/prestontimmons/graphitejs
http://dashboarddude.com/blog/2013/01/23/dashboards-for-graphite/
37. COST
Optimize for adoption rates in your organization by
eliminating cost as a constraint
http://www.examiner.com/images/blog/wysiwyg/image/money].jpg
39. APACHE
Use mod_statsd to capture stats
directly from the Apache request
http://kaleidos.net/files/images/apache318x260.png
http://httpd.apache.org/
https://github.com/jib/mod_statsd
50. PUPPET
Use the Puppet module graphite-report to send Puppet
reporting data directly to Graphite
http://docs.puppetlabs.com/guides/reporting.html
https://github.com/krux/puppet-module-graphite-report