A high-level view of container monitoring and its challenges in data sourcing (short-lived connections) and viz (hairballs). All in the context of Kubernetes, Prometheus, and Weave Cloud

1. Monitoring Containers
David Kaltschmidt
@davkals

3. Rogue waves present
considerable danger for
several reasons:
• unpredictable
• may appear suddenly
or without warning
• and can impact with
tremendous force.

4. Performance Methodologies
• For system engineers
- ways to analyse unfamiliar systems
• For app developers
- guidance for metric and dashboard
design
- Brendan Gregg’s Systems Methodology

5. Traffic Light Anti-Method
1. Turn all metrics into traffic lights
2. Everything green?
No worries, mate.
- Brendan Gregg’s Systems Methodology
🚦

6. https://github.com/weaveworks/scope
Intuition
Engineering:
we need a tool
that gives us an
intuitive
understanding of
the entire system
at a glance.

7. http://cloud.weave.works/
Weave Cloud
Hosted version
of Weave Scope
Runs on K8s

8. DEMO

9. Great, but…
• Short-lived connections
• Hairballs, and shifting layouts

10. scope-probescope-probe
scope-app
Browser
scope-probe
Host 1 Host 2 Host 3
Scope OSS Architecture

11. Connection Tracking
/home/weave # conntrack -E
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41066 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=410
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=36236 dport=32778 src=172.17.0.8 dst=192.168.99.100 sport=80
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41068 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=410
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52996 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41070 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=410
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52998 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41072 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=410
[DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=57975 dport=32777 src=172.17.0.7 dst=192.168.99.100 sport=80
[DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41074 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=410
/home/weave # cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 16810 1 ffff8800d79c1800 100 0 0 1
1: 0100007F:EB74 0100007F:0FC8 06 00000000:00000000 03:0000016D 00000000 0 0 0 3 ffff8800ae3f6e80
2: 0100007F:EB69 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 307011 1 ffff8800cf467040 21 4 30
3: 0100007F:EB7B 0100007F:0FC8 06 00000000:00000000 03:00000D27 00000000 0 0 0 3 ffff8800d7a47538
4: 0100007F:EB7C 0100007F:0FC8 06 00000000:00000000 03:0000110E 00000000 0 0 0 3 ffff8800cf656c70
5: 0100007F:EB67 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 306868 1 ffff8800d79c1040 21 4 27
6: 0100007F:EB76 0100007F:0FC8 06 00000000:00000000 03:00000556 00000000 0 0 0 3 ffff8800d37ac748
7: 0100007F:EB7F 0100007F:0FC8 06 00000000:00000000 03:000014F7 00000000 0 0 0 3 ffff8800d87f0c70

12. Stop sampling,
start listening
EBPF
• user-defined sandboxed kernel
programs
• live-instrumentation on vanilla kernel
• listen to connection events same
way as conntrack does, but with
PID(!)

14. Have you used D3.js?

15. Heuristic I: Alignment

16. Heuristic II: Edge crossings

17. Heuristic III: Commensurate
layout changes

18. Back to
Monitoring
Containers

19. USE vs RED
USE: For every resource, check:
• Utilisation
• Saturation
• Errors
RED: For every service, check:
• Request Rate
• Error rate
• Duration (latency distribution)
* http://www.brendangregg.com/usemethod.html

20. New data sources
• plugins can add
metadata and metrics
• EBPF (ongoing)
• custom
instrumentation

22. Prometheus & K8s
• Kubernetes is already
instrumented for
Prometheus
• Application-level
metrics from
instrumentation

23. Hosted
Prometheus,
multi-tenant
(OSS)
Got a version
running in
Weave Cloud
Run local
Prometheus with
a remote
destination
https://github.com/weaveworks/prism

24. We’re hiring!
London BerlinSan Francisco

25. Questions?
David Kaltschmidt
@davkals https://weave.works