This document discusses the challenges of monitoring microservices and containers. It provides six rules for effective monitoring: 1) spend more time on analysis than data collection, 2) reduce latency of key metrics to under 10 seconds, 3) validate measurement accuracy, 4) make monitoring more available than services monitored, 5) optimize for distributed cloud-native applications, 6) fit metrics to models to understand relationships. It also examines models for infrastructure, flow, and ownership and discusses speed, scale, failures, and testing challenges with microservices.

1. Monitoring Microservices &
Containers: A Challenge
Adrian Cockcroft @adrianco
Technology Fellow - Battery Ventures
May 2015

2. Monitoring
!
Update of my monitoring rules from Monitorama 2014

3. Rule #1: Spend more time working on code
that analyzes the meaning of metrics, than
code that collects, moves, stores and
displays metrics.

4. Rule #2: Metric to display latency needs to
be less than human attention span (~10s)

5. Rule #3: Validate that your measurement
system has enough accuracy and precision.
Collect histograms of response time.

6. Rule #4: Monitoring systems need to be
more available and scalable than the
systems being monitored.

7. Rule #5: Optimize for distributed,
ephemeral, cloud native, containerized
microservices.

8. Rule #6: Fit metrics to models to understand
relationships. (New rule)

10. Container
Instance
e.g. Machine
failure affects
all instances
and containers
inside itZone/DC
Region
Microservice
Model Infrastructure as a
Containment Hierarchy
Machine
Many tools use a naming scheme to imply this model, but
most can’t reason about the relationships

12. Request
Model Applications and Networks
as a Dataflow Graph
APM Tools often model these as business transactions
Microservice Zone/DC
Region

13. Developer Developer
Model Deployment Ownership
and Support
Developer Developer

14. Developer Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer

15. Developer Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer
Monitoring
Tools

16. DeveloperDeveloper Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer
Monitoring
Tools

17. DeveloperDeveloper Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer
Site
Reliability
Monitoring
Tools
Availability
Metrics
99.95% customer
success rate

18. DeveloperDeveloper Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer
Manager Manager
Site
Reliability
Monitoring
Tools
Availability
Metrics
99.95% customer
success rate

19. DeveloperDeveloper Developer
Model Deployment Ownership
and Support
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Micro
service
Developer Developer
Manager Manager
VP
Engineering
Site
Reliability
Monitoring
Tools
Availability
Metrics
99.95% customer
success rate

20. Infrastructure, flow and ownership models
are orthogonal and need to be linked to
make sense of the metrics

21. Monitoring Rules by @adrianco
1. Spend more time on analysis than data collection and display
2. Reduce key business metric latency to less than 10s
3. Validate your measurement system, use histograms
4. Be more available and scalable than the services being monitored
5. Optimize for distributed, ephemeral cloud native applications
6. Fit metrics to models to understand relationships

22. Microservices

23. Microservices
@ideavist

24. A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts

25. A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts
If every service has to be
updated at the same time
it’s not loosely coupled

26. A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts
If every service has to be
updated at the same time
it’s not loosely coupled
If you have to know too much about surrounding
services you don’t have a bounded context. See the
Domain Driven Design book by Eric Evans.

27. Complexity

28. Monolithic apps have unlimited invisible
internal dependencies
!
Vastly more complex than explicit visible
microservice dependencies

29. Speed

30. Speeding Up Deployments
Datacenter Snowflakes
• Deploy in months
• Live for years

31. Speeding Up Deployments
Datacenter Snowflakes
• Deploy in months
• Live for years
Virtualized and Cloud
• Deploy in minutes
• Live for weeks

32. Speeding Up Deployments
Datacenter Snowflakes
• Deploy in months
• Live for years
Virtualized and Cloud
• Deploy in minutes
• Live for weeks
Container Deployments
• Deploy in seconds
• Live for minutes/hours

33. Speeding Up Deployments
Datacenter Snowflakes
• Deploy in months
• Live for years
Virtualized and Cloud
• Deploy in minutes
• Live for weeks
Container Deployments
• Deploy in seconds
• Live for minutes/hours
AWS Lambda Events
• Respond in milliseconds
• Live for seconds

34. Speeding Up Deployments
Measuring CPU usage once a minute makes no sense for containers…
Coping with rate of change is a big challenge for monitoring tools.
Datacenter Snowflakes
• Deploy in months
• Live for years
Virtualized and Cloud
• Deploy in minutes
• Live for weeks
Container Deployments
• Deploy in seconds
• Live for minutes/hours
AWS Lambda Events
• Respond in milliseconds
• Live for seconds

35. Scale

36. A Possible Hierarchy
Continents
Regions
Zones
Services
Versions
Containers
Instances
How Many?
3 to 5
2-4 per Continent
1-5 per Region
100’s per Zone
Many per Service
1000’s per Version
10,000’s
It’s much more challenging
than just a large number of
machines

37. Flow

38. Some tools can show
the request flow
across a few services

39. But interesting
architectures have a
lot of microservices!
Flow visualization is
a challenge.
See http://www.slideshare.net/LappleApple/gilt-from-monolith-ruby-app-to-micro-service-scala-service-architecture

40. Failures

41. ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash Data
Access Layer
Priam Cassandra
Datastore
Simple NetflixOSS
style microservices
architecture on three
AWS Availability Zones

42. ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash Data
Access Layer
Priam Cassandra
Datastore
Simple NetflixOSS
style microservices
architecture on three
AWS Availability Zones

43. ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash Data
Access Layer
Priam Cassandra
Datastore
Simple NetflixOSS
style microservices
architecture on three
AWS Availability Zones
Zone partition/failure
What should you do?
What should monitors show?

44. ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash Data
Access Layer
Priam Cassandra
Datastore
Simple NetflixOSS
style microservices
architecture on three
AWS Availability Zones
Zone partition/failure
What should you do?
What should monitors show?
By design, everything works
with 2 of 3 zones running.
This is not an outage, inform
but don’t touch anything!
Halt deployments perhaps?

45. ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash Data
Access Layer
Priam Cassandra
Datastore
Simple NetflixOSS
style microservices
architecture on three
AWS Availability Zones
Zone partition/failure
What should you do?
What should monitors show?
By design, everything works
with 2 of 3 zones running.
This is not an outage, inform
but don’t touch anything!
Halt deployments perhaps?
Challenge: understand and
communicate common
microservice failure patterns.

46. Testing

47. Testing monitoring tools at scale
gets expensive quickly…

48. Simulation

49. Simulated Microservices
Model and visualize microservices
Simulate interesting architectures
Generate large scale configurations
Eventually stress test real tools
!
See github.com/adrianco/spigo
Simulate Protocol Interactions in Go
Visualize with D3
ELB Load Balancer
Zuul API Proxy
Karyon
Business
Logic
Staash
Data
Access
Layer
Priam Cassandra
Datastore
Three
Availability
Zones

50. netflixoss.go architecture
!!!!!!!!!asgard.Create(cname, asgard.PriamCassandraPkg, regions, priamCassandracount, "eureka", cname)
asgard.Create(tname, asgard.StaashPkg, regions, staashcount, cname)
asgard.Create(jname, asgard.KaryonPkg, regions, javacount, tname)
asgard.Create(nname, asgard.KaryonPkg, regions, nodecount, jname)
asgard.Create(zuname, asgard.ZuulPkg, regions, zuulcount, nname)
asgard.Create(elbname, asgard.ElbPkg, regions, 0, zuname)
asgard.Run(asgard.Create(dns, asgard.DenominatorPkg, 0, 0, elbname), jname) // victimize a javaweb
Tooling
New tier
name
Tier
package
Region
count: 1
Node
count
List of tier
dependencies

51. Run and log results to json
$ spigo -a netflixoss -d 10 -j
2015/05/21 00:05:32 netflixoss: scaling to 100%
2015/05/21 00:05:32 netflixoss.edda: starting
2015/05/21 00:05:32 netflixoss.us-east-1.zoneA.eureka.eureka.eureka0: starting
2015/05/21 00:05:32 netflixoss.us-east-1.zoneB.eureka.eureka.eureka1: starting
2015/05/21 00:05:32 netflixoss.us-east-1.zoneC.eureka.eureka.eureka2: starting
2015/05/21 00:05:32 netflixoss.*.*.www.denominator.www0 activity rate 10ms
2015/05/21 00:05:37 chaosmonkey delete: netflixoss.us-east-1.zoneC.javaweb.karyon.javaweb14
2015/05/21 00:05:42 asgard: Shutdown
2015/05/21 00:05:42 netflixoss.us-east-1.zoneB.eureka.eureka.eureka1: closing
2015/05/21 00:05:42 netflixoss.us-east-1.zoneA.eureka.eureka.eureka0: closing
2015/05/21 00:05:42 netflixoss.us-east-1.zoneC.eureka.eureka.eureka2: closing
2015/05/21 00:05:42 spigo: complete
2015/05/21 00:05:42 netflixoss.edda: closing
10 sec
run time
edda.go logs
config to json
eureka.go
service
registry per
zone
Chaos
monkey
victim!

52. Simianviz from json logs
http://simianviz.divshot.io/netflixoss/1
ELB splits
traffic over
zones in
single region
microservices
Cassandra
Cluster
Six regions
Big thanks to @kurtiskemple

53. Why Build Spigo?
Generate test microservice configurations at scale
Stress monitoring tools and simulated game day training
!
Eventually (i.e. not implemented yet)
Dynamically vary configuration: autoscale, code push
Chaos gorilla for zone, region failures and partitions
Websocket connection between spigo and simianviz display
!

54. My challenge to you:
Build your architecture in Spigo.
Stress monitoring tools with it.
Help fix monitoring for microservices!
!
@mgroeniger

55. Questions?
Disclosure: some of the companies mentioned may be Battery Ventures Portfolio Companies
See www.battery.com for a list of portfolio investments
● Microservices Challenges
● Speed and Scale
● Flow and Failures
● Testing and Simulation
!
● Battery Ventures http://www.battery.com
● Adrian’s Tweets @adrianco and Blog http://perfcap.blogspot.com
● Slideshare http://slideshare.com/adriancockcroft
● Github http://github.com/adrianco/spigo

56. What does @adrianco do?
@adrianco
Technology Due
Diligence on Deals
Presentations at
Conferences
Presentations at
Companies
Technical Advice
for Portfolio
Companies
Program
Committee for
Conferences
Networking with
Interesting PeopleTinkering with
Technologies
Maintain Deep
Relationship with
Cloud Vendors

57. | Battery Ventures
Portfolio Companies for Enterprise IT
Security
Visit http://www.battery.com/our-companies/ for a full list of all portfolio companies in which all Battery Funds have invested.
Palo Alto Networks
Enterprise IT
Operations &
Management
Big DataCompute
Networking
Storage