How to Monitor Kubernetes, the 5 layers that really matter

1. Jorge Salamero Sanz
@bencerillo @sysdig
The Top 5
Kubernetes Metrics
to Monitor

2. @bencerillo @sysdig
Scope of monitoring
Monitoring is for Sysadmins or DevOps
APM is for developers only

3. @bencerillo @sysdig
Scope of monitoring
Monitoring is for Sysadmins or DevOps
APM is for developers only
Developers run and monitor their own services

4. @bencerillo @sysdig
New layers
Services / containers not coupled with nodes
Understanding Kubernetes metadata is an absolute requirement
How Kubernetes changes monitoring

5. @bencerillo @sysdig
1. Application
2. Services
3. Kubernetes deployment
4. Kubernetes internals
5. Host / node
The 5 layers to monitor in Kubernetes

6. @bencerillo @sysdig
• Is your application running properly?
• Service availability, response time, requests (and errors) per second
• HTTP response time:
net.http.request.time > 500ms
1. Application layer

7. @bencerillo @sysdig
• Are my services in Kubernetes working properly?
• HTTP requests, database connections and replication, threads and connections, middleware:
uwsgi workers, JVM heap, etc
• Connections per second:
[db].net.connections < 2 conn/s
2. Service layer

8. @bencerillo @sysdig
• Enough pods/containers running for each app?
• Or any at all? Any pods in CrashLoopBackOff?
• The holy check:
kubernetes.replicaSet.replicas.running < kubernetes.replicaSet.replicas.desired
3. Kubernetes infrastructure

9. @bencerillo @sysdig
• Are Kubernetes internals working?
• Are nodes & etcd running? API? kubelet and Docker? overbooked nodes?
• Do we have enough nodes in the cluster?
kube_node_status_ready > x
Kube_node_spec_unschedulable < y
4. Kubernetes internals

10. @bencerillo @sysdig
• Are the node resources available?
• Is host up or down/unreachable?
• Disk usage:
fs.used.percent > 90%
5. Host / node

11. @bencerillo @sysdig
1. Application
2. Services
3. Kubernetes deployment
4. Kubernetes internals
5. Host / node
The 5 layers to monitor in Kubernetes

12. Thank You!
@bencerillo
@sysdig
sysdig.com | sysdig.org