At the end of DockerCon 2015, Dieter Reuter from Hypriot presented a demo running 500 Docker containers on a Raspberry Pi 2 device but he knew that number could be at least doubled. TL;DR Dieter was right. A big congratulations to Damien Duportal, Nicolas de Loof and Yoann Dubreuil for running 2500 web servers in containers on a single Raspberry Pi 2! Read more: http://blog.docker.com/2015/10/raspberry-pi-dockercon-challenge-winner/

1. Docker RPi2 challenge
@damienduportal @ndeloof @YoannDubreuil @Quintus23M

2. Challenge Introduction
DockerCon 15, June 22-23
Live Demo: starting 100x Docker Containers on a Raspberry Pi 2 B
showed a screen shot with 250x running Containers
and plenty of resources/memory left, let’s improve it

4. OK, let’s start the Challenge
Challenge: How many Containers could be really started on a Pi 2?
my wish: I’d like to see 1000x Containers
personal record to this time: 613x Containers
--- but honestly, I was not sure, if this is possible at all!

5. let’s break the challenge

6. First tries to the record
Try, Learn and Share !

7. 1 - Measure and automate all the things
Measures :
sysstat for post mortem
node-collector from Prometheus.io for “real time”
Provisionning :
Basic shell script published on my Github
Let’s try Ansible later : easier to maintain, need to learn :)

9. 2 - Lower the container footprint
Tried with custom compiled nginx for ARM with few extensions
➢ ~ 80 containers
Footprint too big per container. Reading carefully Hypriot Blog : "rpi-nano-
httpd" : 16 Kb footprint, 88 Kb image
➢ ~150 containers

10. 3 - Docker tuning
Wanted to disable tcp-proxy process : no use here
Docker logs says “iptables” errors
Footprint with iptable + net. namespace : around 1 Mb !
Disable network / port forwarding + --net=host (Client side)
--bridge=none --iptables=false --ipv6=false --ip-
forward=false --ip-masq=false -tls=false --userland-
proxy=false (Engine side)

11. 4 - Moaaar Docker tuning
No debug : --debug=true
No filesystem writing : --read-only=true
No logging : --log-driver=none
Following Docker upgrades :
--ipc=host
--utc=host

12. 5 - Linux tuning : start sharing
Tune sysctl : limits ! (MaxOpenFiles, MaxProcs, etc.)
Still stuck, footprint / container still big (~ 750 Kb)
Help from @booya and Hypriot guys + eading the /etc/init.d/docker script :
Stack size (8 Mb -> 2 Mb in Hypriot:Jack)
Playing around doing abacus :
<32 Kb breaks docker pull, <16Kb breaks docker run
~ 1700 containers

13. 6 - To the record : Learning and sharing
GOGC to 10
~ 2 200 containers
Docker Hack days + Twitter :
KSM : KO (need to recompile Kernel)
Learning the Linux memory management : Freeing caches + disabling OOM
~ 2498 containers !!!

14. Share and challenge !
Community, discussions, challenge and Whales

15. BreizhCamp Hack (night) Party

16. Brainstorm for ideas
… then test everything in arbitrary order
help each other
drink some beer
learn a lot
Our Methodology

17. where to start?
first naïve try
only 38 containers :- !
but 70 on a RPi1
figure out RPi2 limits without Docker
web server footprint
network namespace footprint
get some help !

18. web server
from hypriot/rpi-nano-httpd image
written in ARM assembly code
already highly optimized
1 page for code
1 page for data
1 page for stack
1 page for vsdo

19. network namespace RPi2 limit
launched web server in a dedicated network namespace
ip netns exec <NS_NUMBER> httpd
RPi2 limit is ~ 1.100 network namespace
=> To break the challenge, we needed to run without network isolation

20. network configuration
disabled everything we could on Docker daemon
--bridge=none --userland-proxy=false --sig-proxy=false
--ipv6=false --ip-masq=false --iptables=false
started containers with --net=host
disabled IPv6 on Linux kernel boot with ipv6.disable=1

21. Systemd
Docker daemon run as root
… but still has some limits set by systemd (so the 38 containers...)
LimitSIGPENDING=infinity
LimitNOFILE=infinity
LimitAS=infinity
LimitNPROC=infinity
LimitSTACK=?

22. stack size
Damien pro-tip : adjust LimitSTACK parameter to your needs
Default stack size is 8Mb
a stack consume 8Mb of process VM space (8 * 4 * 38 = 1,2 Gb)
=> ~ 1800 / 2000 containers

23. process memory layout
RPi2 processor is 32 bits !
32 bit process address space
only 3Gb of virtual memory per process
collision between stack and heap memory areas
reduce thread stack size
smallest working thread stack size: 24kb

24. speed up tests
launching thousands of containers on a RPi2 takes hours if not days!
everything in memory with zram devices
swap (ratio 5:1)
/var/lib/docker on ext4 FS (ratio 10:1)
swap as early as possible to keep free memory (vm.swappiness = 100)
USB external disk vs low perf, I/O limited SD card

25. system tuning
limit memory consumption
reduce GPU memory to 16Mb (can’t do less)
blacklisted non required Linux modules
remove some Linux limits
vm.overcommit = 1
kernel.pid_max = 32768
kernel.threads-max = 14812

26. Challenge
completed!
We started 2499containers !
RAM on RPi2 was not exhausted but Docker daemon crashed
docker[307]: runtime: program exceeds 10000-thread limit

27. Why is there a limit?
4 threads per container
10.000 threads for a Go application => 2500 containers max
Unlocked this with runtime.debug.SetMaxThread(12000)
hack not eligible for RpiDocker challenge, was just to confirm
can run ~2740 webserver containers, before actual OOM

28. what did not work
Btrfs
not working properly : strange web server 404 failures after ~20
successful launchs
stick with overlayfs
LXC driver
way sloooooooower

29. What’s next
Understand why Docker do need 4 threads per container
(hey, lot’s of Docker core contributors here, time to ask !)

30. Collaboration (and beer)
were the keys
to break this challenge !

31. links
Nicolas’ blog
http://blog.loof.fr/2015/10/how-to-run-2500-webservers-on-raspberry.html
code
https://github.com/ndeloof/RPiDockerKillerScore