How ovh went from 20 to 2500 containers in production

1. Scaling docker @OVH
Timothée GERMAIN devops@OVH
From 20 to 2500 running containers

2. About me
Timothée GERMAIN
Devops for Containers team @ OVH since 2015
Work with containers since 2014
Twitter : DarkNihilius1

3. Docker @ OVH
● 8 clusters
○ Internal production : 5
○ External beta/gamma : 2
○ Preproduction : 1
● 2500 running containers
● 800 Hosts
● 3000 cpu cores
● 12TB RAM
● 2 docker registry (internal & external)
● 100s of deployment per day
● 5 people

4. The beginning : 1 - 20 containers
Everything fit on 1 host
+ Easy to deploy docker run
+ Easy to build docker build
+ Easy to debug docker exec/logs/stats
- No high availability
- Not scalable

5. Scaling up : +20 containers, 1 team
You need an orchestrator :
● Docker swarm (Docker inc.)
● Mesos & Marathon (Apache Software Foundation)
● Kubernetes (Google then CNCF)
Don’t fit on 1 host ?
Need high availability ?

6. Mesos & marathon in 60 seconds
Linux kernel : provide abstraction of
hardware
Linux kernel
Your application
Linux scheduler
Scheduler : say which application run
when with which resources

7. Mesos & marathon in 60 seconds
Mesos : provide abstraction of
multiple hosts
Mesos
Your (Docker)
application
Framework
Framework (Marathon) : say which
application run when with which resources

8. Needs :
● Isolate team/client
● Be cost efficient, mutualize what we can
Scaling up : 20+ containers, 2+ team
More team !

9. Scaling up : 20+ containers, 2+ team
Marathon : no multitenancy out of the box
The solutions we try :
1) One marathon per team
2) One marathon with logical isolation (middleware)
3) One mesos & marathon per team

10. Multitenancy : One marathon per team
+ Good isolation
+ Mesos mutualization
+ Fair blast radius
- Does not scale well
Mesos
Marathon B
Marathon CMarathon A

11. Multitenancy : One marathon and logical isolation
+ Scale
+ Give us control on user input
- Less isolation
- Lots of maintenance
- Hard to keep up to date with
marathon API
- Big blast radius
Mesos
Marathon
Middleware

12. Multitenancy : One marathon & mesos cluster per team
+ Scale even better
+ Little blast radius
+ Best isolation
+ Modular
- Less/no mutualisation
- Costly
- Lot of parts to deploy and
maintain
Mesos B
Marathon B
Mesos A
Marathon A

13. Deploy mesos and marathon
using mesos and marathon
Lot of parts to deploy and
maintain ?
we already have a tool for that :
Mesos and Marathon !
Mesos B
Marathon B
Mesos A
Marathon A
Mesos
Admin
Marathon
Admin

14. Questions ?
Thanks for your time

15. Log and metrics : 1 - 20 containers
Docker stats
Docker logs
Direct calls on docker daemon

16. Log and metrics : +20 containers, 2+ team
Not docker logs nor docker stats