Stream Processing Inside Librato [Monitorama PDX 2015]
As modern software engineering tools and techniques continue to evolve, very few engineering teams have been able to meaningfully improve their operational visibility. In this talk Librato's Dave Josephsen makes that case that monitoring has always been a signal processing problem. He explains in practical terms what that means, and how viewing the problem in the context of signal processing necessitates a different sort of toolset. He then shares Librato's stream processing infrastructure, which is an implementation of a signal-processing telemetry system, and shows how our infrastructure solves many of the data persistence and signal transformation problems that engineering teams who use traditional operations-centric monitoring toolchain have struggled with for years.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (19)
Destacado
Destacado (20)
Similar a Stream Processing Inside Librato [Monitorama PDX 2015]
Similar a Stream Processing Inside Librato [Monitorama PDX 2015] (20)
Último
Último (20)
Stream Processing Inside Librato [Monitorama PDX 2015]
- 1. hi.
- 4. Who put Stream Processing in my Monitoring Conference?
- 5. Let’s talk about signal processing
- 18. Nagios Zabbix Collectd DAC (transform and normalize the signal) (e.g., translate input to a standard json blob)
- 19. Zabbix Filters (de-duplication, noise reduction) (e.g., compute percentiles to filter outliers)
- 20. Multiplex (aggregation) (e.g., aggregate http 200’s from multiple hosts)
- 21. Effects (ad-hoc external signal processing) ??
- 22. Effects (ad hoc external signal processing) ??
- 29. Disk Latency Queue Latency Model A Model B
- 31. ing MultiVariate Data Science Menu Specials Gaussian Fuzzy Discrete Hybridized Smoothing ExponentialSmoothing Holt-Winters Chicken Fried Exponential Smoothing Naieve Baysian linear regression Markov Chains extreme studentized Kung-Pow distance vector grouping stream decompisiton Organic, free-range Ansecomb Quartet really really smooth Exponential Smoothing xtra seasonality available on request Downsampling Statistical Convergence Lacunary Sequence Hashsum Distance Lambda Convergence Lambda ANYTHING Seasonal Hybridized Whatever HALF OFF! Anything Named after a turn-of-the-century Member of the Russian Intelligencia Spacial Forecasting deviant Test Quantile isolation Probablistic merging Hybridized/Fuzzy Combo platter! Collaborative n-gram cosine matching Robust Sequential Map-Reduce Micro-lot N-variate chi-square Ask about our locally-sourced quantile robustness Roasted Daily!
- 33. Scientist gonna science Engineer gonna.. yeah.
- 34. Scientist gonna scienceData Engineer gonna ??
- 35. Making a model is just the beginning
- 43. Nagios
- 44. GraphiosNagios
- 47. raphios Statsd Graphite Grafana
- 48. raphios Statsd Graphite Grafana Math thingy ???
- 49. Effects (ad hoc external signal processing) ??
- 50. Stuff Statsd Graphite Grafana
- 52. Stuff Graphite GrafanaHeka Model A Model B
- 53. Stuff Graphite GrafanaHeka Model A Model B carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay
- 54. Stuff Graphite GrafanaHeka Model A Model B carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay carbon relay Elastisearch Elastisearch
- 59. Data in Truth out
- 62. X
- 63. XX X
- 64. XX X X X X X
- 65. Kafka
- 66. SSA Kafka
- 67. SSA Kafka
- 68. SSA r60 Kafka
- 70. SSA
- 78. SSA MySQL
- 79. SSA Alerts MySQL
- 80. SSA Alerts MySQL
- 82. SSA Alerts MySQL
- 83. Storm is “pretty OK”
- 84. Lacks visibility into flow, performance, and failures
- 85. Netty? JVM Hops? Ser/Der?
- 86. Samza? Spark? NIH? ( kafka + zk + mezos + disruptor+? )
- 87. Questions?