There is a common belief that in order to solve more [advanced] alerting cases and get more complete coverage, we need complex, often math-heavy solutions based on machine learning or stream processing. This talk sets context and pro's/cons for such approaches, and provides anecdotal examples from the industry, nuancing the applicability of these methods. We then explore how we can get dramatically better alerting, as well as make our lives a lot easier by optimizing workflow and machine-human interaction through an alerting IDE (exemplified by bosun), basic logic, basic math and metric metadata, even for solving complicated alerting problems such as detecting faults in seasonal timeseries data. https://www.usenix.org/conference/srecon16europe/program/presentation/plaetinck

1. Next-generation
alerting and fault detection
Dieter Plaetinck, raintank
SRECon16 Europe – Dublin, Ireland July 12, 2016

2. Alerting, fault & anomaly detection through:
Machine learning
event & stream processing
Alerting IDE’s

7. Also on
● support for Graphite and Grafana

8. Also on
● support for Graphite and Grafana
● hosted Graphite and Grafana

9. Presumptions
● Monitoring using metrics in place

10. Presumptions
● Monitoring using metrics in place
● Alerting on metrics

11. Presumptions
● Monitoring using metrics in place
● Alerting on metrics
● Alerts need high signal/noise ratio

17. www.quora.com/unanswered

18. Google trends

19. Static thresholds → automated anomaly detection
● Not scaling / too much data

20. Static thresholds → automated anomaly detection
● Not scaling / too much data
● Infrastructure complexity

21. Static thresholds → automated anomaly detection
● Not scaling / too much data
● Infrastructure complexity
● Alerting on Patterns

22. Machine learning is a subfield of computer science that evolved from the study of pattern
recognition and computational learning theory in artificial intelligence. In 1959, Arthur Samuel defined
machine learning as a field of study that
gives computers the ability to learn without being explicitly
programmed.
Machine learning explores the study and construction of
algorithms that can learn from and make predictions on data.
Such algorithms operate by building a model from an example training set of input observations in order
to make data-driven predictions or decisions expressed as outputs, rather than following strictly static
program instructions.”
https://en.wikipedia.org/wiki/Machine_learning

23. http://www.extremetech.com/extreme/224445-its-2-0-how-googles-deepmind-is-beating-the-best-in-
go-and-why-that-matters

24. https://research.googleblog.com/2014/09/building-deeper-understanding-of-images.html

26. Using machine learning
for automated anomaly detection

29. Challenges.

31. Challenges
1 ● e.g. Amazon, Facebook, LinkedIncontext

32. 1 ● e.g. Amazon, Facebook, LinkedIn
● e.g. infrastructure change
context
Challenges

33. 2 ● Games vs your infraChanging rules
Challenges

34. Challenges
2 ● Games vs your infra
● Trained model doesn’t work
on new scenarios
Changing rules

35. 3 Image recognition, security
vs
ops metrics
Signal strength
Challenges

36. 4 ● e.g. super fast to fastrelevancy
Challenges

37. 4 ● e.g. super fast to fast
● e.g. redundancy failover
relevancy
Challenges

38. 4 ● e.g. super fast to fast
● e.g. redundancy failover
operator knows best
relevancy
Challenges

39. 5 ● data prep: filtering, selection, cleaning
● statistical modeling, model selection
● training, testing
● track performance & maintenance
● operate infrastructure
● fitting UX/UI
Effort
Challenges

40. 6 ● IntrinsicComplexity
Challenges

41. 6 ● Intrinsic
● Incidental
https://engineering.quora.com/Avoiding-Complexity-
of-Machine-Learning-Systems
Complexity
Challenges

42. ML / AD for operations
has merits
BUT:

43. ML / AD for operations
has merits
BUT: ● Anomalies != Faults. Signal/noise trap

44. ML / AD for operations
has merits
BUT: ● Anomalies != Faults. Signal/noise trap
● Significant effort & complexity

45. ML / AD for operations
has merits
BUT: ● Anomalies != Faults. Signal/noise trap
● Significant effort & complexity
● Limited use cases

46. What might help
● Enrich metric metadata (metrics20.org)

47. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context

48. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context
classification for model selection

49. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context
classification for model selection
derive relevancy

50. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context
classification for model selection
derive relevancy
● integration with CM, PaaS.

51. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context
classification for model selection
derive relevancy
● integration with CM, PaaS.
awareness of infrastructure

52. What might help
● Enrich metric metadata (metrics20.org)
clustered, stronger signals with more context
classification for model selection
derive relevancy
● integration with CM, PaaS.
awareness of infrastructure
awareness of infrastructure change

53. HOW do
THEYdo it?

54. https://codeascraft.com/2013/06/11/introducing-kale/

56. https://www.oreilly.com/ideas/monitoring-distributed-systems
“it’s important that monitoring systems - especially the critical
path from the onset of a production problem, through a page
to a human, through basic triage and deep debugging - be kept
simple and comprehensible by everyone on the
team.”
“Similarly, to keep noise low and signal high, the elements of
your monitoring system that direct to a pager need to be very
simple and robust. Rules that generate alerts for humans
should be simple to understand and represent a clear
failure.”

57. Conclusion

59. CEP
& Stream processing

60. CEP & stream processing
e.g. storm, riemann.io, spark streaming

61. in → logic → out

62. Riemann.io

63. CEP & stream processing
Compared to query-based alerting systems:

64. CEP & stream processing
Compared to query-based alerting systems:
● Good scheduling guarantees/execution timeliness

65. CEP & stream processing
Compared to query-based alerting systems:
● Good scheduling guarantees/execution timeliness
● Unfamiliar paradigm (maybe)

66. CEP & stream processing
Compared to query-based alerting systems:
● Good scheduling guarantees/execution timeliness
● Unfamiliar paradigm (maybe)
● Performance/scalability (maybe)

67. CEP & stream processing
Compared to query-based alerting systems:
● Good scheduling guarantees/execution timeliness
● Unfamiliar paradigm (maybe)
● Performance/scalability (maybe)
● operational complexity (maybe)

68. Conclusion

69. Not a bad idea…
But doesn’t get to the root of the alerting problems.

70. Aha!

72. Picture by Matt Simmons

73. IDE for alerting
Support programmers building and maintaining software

74. IDE for alerting
Support programmers building and maintaining software
Support operators building and maintaining alerting

79. 1
vs traditional alerting, machine learning
[historical] testing
Key features

80. 2 Arbitrary scopedata juggling
Key features

81. Key features
2 Arbitrary scope
Arbitrary data
data juggling

82. 3dependencies
Key features

83. http://www.slideshare.net/adriancockcroft/gluecon-monitoring-microservices-and-containers-a-challenge

84. Key features
4transcience

85. Key features
5DRY

86. Key
insights.

87. Key insights
1remove hassle wrt improving signal/noise

88. Key insights
1
● ongoing maintenance & tuning is critical
remove hassle wrt improving signal/noise

89. Key insights
1
● ongoing maintenance & tuning is critical
● code for UI and logic > knobs
remove hassle wrt improving signal/noise

90. Key insights
1
● ongoing maintenance & tuning is critical
● code for UI and logic > knobs
● leveraging additional data
remove hassle wrt improving signal/noise

91. Key insights
2 ● Author to recipientcommunication

92. Key insights
2 ● Author to recipient
● Alert often primary UI
communication

93. Key insights
3Human > computer

94. Key insights
4attention is scarce, expensive

95. Key insights
4attention is scarce, expensive
"provide monitoring platform that enables operators to
efficiently utilize their attention"

96. fault detection
with bosun

97. Classify series & find KPI’s

98. Smoothly seasonal: good

99. Smoothly seasonal: offset

100. Smoothly seasonal: spikes

101. Smoothly seasonal: erratic

102. Band(), graphiteBand()
bosun.org/expressions.html
Solution 1/2 : strength

103. Solution 1/2 : strength

104. Solution 2/2 : erraticness

105. Deviation-now
Erraticness now = --------------------------
Deviation-historical
Solution 2/2 : erraticness

106. Deviation-now median-historical
Erraticness now = -------------------------- * -----------------------
Deviation-historical median-now
Solution 2/2 : erraticness

107. deviation-now * median-historical
Erraticness now = -------------------------------------------------------
(deviation-historical * median-now) + 0.01
Solution 2/2 : erraticness

112. dieter.plaetinck.be/post/practical-fault-detection-on-timeseries-part-2
More details
Bosun macro, template & example
Grafana dashboard

113. Static thresholds → automated anomaly detection
● Not scaling / too much data

114. Static thresholds → automated anomaly detection
● Not scaling / too much data
● Infrastructure complexity

115. Static thresholds → automated anomaly detection
● Not scaling / too much data
● Infrastructure complexity
● Alerting on patterns

116. Conclusion

117. ● All about the workflow

118. ● All about the workflow
● An IDE like bosun exponentially boosts ability to maintain high
signal/noise alerting

119. ● All about the workflow
● An IDE like bosun exponentially boosts ability to maintain high
signal/noise alerting
● Build & share!

120. Want more ?
● bosun.org/resources presentations by Kyle Brandt (LISA 2014 + Monitorama 2015)
● “my philosophy on alerting” by Rob Ewaschuk
● kitchensoap.com/2015/05/01/openlettertomonitoringproducts
● kitchensoap.com/2013/07/22/owning-attention-considerations-for-alert-design
● “monitoring microservices” by Adrian Cockroft
● (dieter.plaetinck.be/post/practical-fault-detection-alerting-dont-need-to-be-data-
scientist)
● dieter.plaetinck.be/post/practical-fault-detection-on-timeseries-part-2
● metrics20.org/media
● mabrek.github.io
● iwringer.wordpress.com
@Dieter_be - @raintanksaas – slack.raintank.io – raintank.io – bosun.org – grafana.org