Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
FROM MONOLITH TO DOCKER
DISTRIBUTED APPLICATIONS
Carlos Sanchez
@csanchez
ABOUT ME
Senior So ware Engineer @ CloudBees
Author of Jenkins Kubernetes plugin
Long time OSS contributor at Apache Maven...
Linux containers
Filesystem
Users
Processes
Network
BUT IT IS NOT TRIVIAL
OUR USE CASE
Scaling Jenkins
Your mileage may vary
ARCHITECTURE
Docker Docker Docker
Isolated Jenkins masters
Isolated slaves and jobs
Memory and CPU limits
How would you design your infrastructure if
you couldn't login? Ever.
Kelsey Hightower
EMBRACE FAILURE!
CLUSTER SCHEDULING
Distribute tasks across a cluster of hosts
Running in public cloud, private cloud, VMs or bare metal
HA...
APACHE MESOS
A distributed systems kernel
ALTERNATIVES
Docker Swarm / Kubernetes
MESOSPHERE MARATHON
APACHE ZOOKEEPER
TERRAFORM
TERRAFORM
resource "aws_instance" "worker" {
count = 1
instance_type = "m3.large"
ami = "ami-xxxxxx"
key_name = "tiger-csa...
TERRAFORM
State is managed
Runs are idempotent
terraform apply
Sometimes it is too automatic
Changing image id will restar...
STORAGE
Handling distributed storage
Servers can start in any host of the cluster
And they can move when they are restarted
DOCKER VOLUME PLUGINS
Flocker
GlusterFS
NFS
EBS
KUBERNETES
GCE disks
Flocker
GlusterFS
NFS
EBS
SIDEKICK CONTAINER
A privileged container that manages mounting for other
containers
Can execute commands in the host and ...
IN OUR CASE
Sidekick container (castle service)
Jenkins masters need persistent storage, slaves (typically)
don't
Supporti...
CASTLE
Jenkins master container requests data on startup using
entrypoint
REST call to Castle
Castle checks authentication...
CASTLE
Mounts storage in requesting container
EBS is mounted to host, then bind mounted into
container
NFS is mounted dire...
CASTLE: BACKUPS AND CLEANUP
Periodically takes S3 snapshots from EBS volumes in AWS
Cleanups happening at different stages ...
PERMISSIONS
Containers should not run as root
Container user id != host user id
i.e. jenkinsuser in container is always 10...
CAVEATS
Only a limited number of EBS volumes can be mounted
Docs say /dev/sd[f-p], but /dev/sd[q-z]seem to
work too
Someti...
MEMORY
Scheduler needs to account for container memory
requirements and host available memory
Prevent containers for using...
WHAT DO YOU THINK HAPPENS
WHEN?
Your container goes over memory quota?
WHAT ABOUT THE JVM?
WHAT ABOUT THE CHILD
PROCESSES?
CPU
Scheduler needs to account for container CPU requirements
and host available CPUs
WHAT DO YOU THINK HAPPENS
WHEN?
Your container tries to access more than one CPU
Your container goes over CPU limits
Totally different from memory
Mesos/Kubernetes CPU translates into Docker --cpu-
shares
OTHER
CONSIDERATIONS
DOCKER AND THE PID 1 ZOMBIE
REAPING PROBLEM
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-
zombie-reaping-proble...
PROCESS ADOPTION
THIS IS A PROBLEM IN DOCKER
Jenkins slaves run multiple processes
But Jenkins masters too, and they are long running
TINI
Systemd or SysV init is too heavyweight for containers
All Tini does is spawn a single child (Tini is
meant to be run...
PROCESS REAPING
Docker 1.9 gave us trouble at scale, rolled back to 1.8
Lots of defunct processes
NETWORKING
Multiple services running in the same ports
Must redirect from random ports in the host
Services running in one...
NETWORKING: SERVICE DISCOVERY
DNS is not great, caching can happen at multiple levels
marathon-lbuses haproxyand Marathon ...
NETWORKING: SECURITY
Prevent/Allow
from to
container host
container container
container another host
container container i...
NETWORKING: SECURITY
Prevent/Allow
from to
container host iptables
container container --icc=false+ --link,
docker0bridge ...
NETWORKING: SOFTWARE DEFINED
NETWORKS
Create new custom networks on top of physical networks
Allow grouping containers in ...
NETWORKING: SOFTWARE DEFINED
NETWORKS
Battlefield: Calico, Flannel, Weave and Docker Overlay
Network
DOCKER OVERLAY
Docker networking with default overlaydriver, using
VxLAN
# On the Swarm master
docker network create --dri...
WEAVE
UDP and VxLAN backends
COREOS FLANNEL
UDP and VxLAN backends
Uses etcdfor key-value store
PROJECT CALICO
A pure Layer 3 model
SCALING
New and interesting problems
TERRAFORM AWS
Instances
Keypairs
Security Groups
S3 buckets
ELB
VPCs
AWS
Resource limits: VPCs, S3 snapshots, some instance sizes
Rate limits: affect the whole account
Retrying is your friend,...
TERRAFORM OPENSTACK
Instances
Keypairs
Security Groups
OPENSTACK
Custom flavors
Custom images
Different CLI commands
There are not two OpenStack installations that are the same
UPGRADES /
MAINTENANCE
Moving containers from hosts
Draining hosts
Rolling updates
Blue/Green deployment
Immutable infrast...
THANKS
csanchez.org
csanchez
carlossg
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
From Monolith to Docker Distributed Applications
Próxima SlideShare
Cargando en…5
×

From Monolith to Docker Distributed Applications

588 visualizaciones

Publicado el

Docker is revolutionizing the way people think about applications and deployments. It provides a simple way to run and distribute Linux containers for a variety of use cases, from lightweight virtual machines to complex distributed micro-services architectures.
Containers allow to run services in isolation with a minimum performance penalty, increased speed, easier configuration and less complexity, making it ideal for continuous integration and continuous delivery based workloads. But migrating an existing application to a distributed microservices architecture is no easy task, requiring a shift in the software development, networking and storage to accommodate the new architecture.
We will provide insight on our experience creating a Jenkins platform based on distributed Docker containers running on Apache Mesos and Marathon, applicable for all types of applications, but specially Java and JVM based nones.

Publicado en: Software
  • Sé el primero en comentar

From Monolith to Docker Distributed Applications

  1. 1. FROM MONOLITH TO DOCKER DISTRIBUTED APPLICATIONS Carlos Sanchez @csanchez
  2. 2. ABOUT ME Senior So ware Engineer @ CloudBees Author of Jenkins Kubernetes plugin Long time OSS contributor at Apache Maven, Eclipse, Puppet,…
  3. 3. Linux containers Filesystem Users Processes Network
  4. 4. BUT IT IS NOT TRIVIAL
  5. 5. OUR USE CASE Scaling Jenkins Your mileage may vary
  6. 6. ARCHITECTURE Docker Docker Docker Isolated Jenkins masters Isolated slaves and jobs Memory and CPU limits
  7. 7. How would you design your infrastructure if you couldn't login? Ever. Kelsey Hightower
  8. 8. EMBRACE FAILURE!
  9. 9. CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, private cloud, VMs or bare metal HA and fault tolerant With Docker support of course
  10. 10. APACHE MESOS A distributed systems kernel
  11. 11. ALTERNATIVES Docker Swarm / Kubernetes
  12. 12. MESOSPHERE MARATHON
  13. 13. APACHE ZOOKEEPER
  14. 14. TERRAFORM
  15. 15. TERRAFORM resource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
  16. 16. TERRAFORM State is managed Runs are idempotent terraform apply Sometimes it is too automatic Changing image id will restart all instances
  17. 17. STORAGE Handling distributed storage Servers can start in any host of the cluster And they can move when they are restarted
  18. 18. DOCKER VOLUME PLUGINS Flocker GlusterFS NFS EBS
  19. 19. KUBERNETES GCE disks Flocker GlusterFS NFS EBS
  20. 20. SIDEKICK CONTAINER A privileged container that manages mounting for other containers Can execute commands in the host and other containers A lot of magic happening with nsenter
  21. 21. IN OUR CASE Sidekick container (castle service) Jenkins masters need persistent storage, slaves (typically) don't Supporting EBS (AWS) and external NFS
  22. 22. CASTLE Jenkins master container requests data on startup using entrypoint REST call to Castle Castle checks authentication Creates necessary storage in the backend EBS volumes from snapshots Directories in NFS backend
  23. 23. CASTLE Mounts storage in requesting container EBS is mounted to host, then bind mounted into container NFS is mounted directly in container Listens to Docker event stream for killed containers
  24. 24. CASTLE: BACKUPS AND CLEANUP Periodically takes S3 snapshots from EBS volumes in AWS Cleanups happening at different stages and periodically EMBRACE FAILURE!
  25. 25. PERMISSIONS Containers should not run as root Container user id != host user id i.e. jenkinsuser in container is always 1000 but matches ubuntuuser in host
  26. 26. CAVEATS Only a limited number of EBS volumes can be mounted Docs say /dev/sd[f-p], but /dev/sd[q-z]seem to work too Sometimes the device gets corrupt and no more EBS volumes can be mounted there NFS users must be centralized and match in cluster and NFS server
  27. 27. MEMORY Scheduler needs to account for container memory requirements and host available memory Prevent containers for using more memory than allowed Memory constrains translate to Docker --memory
  28. 28. WHAT DO YOU THINK HAPPENS WHEN? Your container goes over memory quota?
  29. 29. WHAT ABOUT THE JVM?
  30. 30. WHAT ABOUT THE CHILD PROCESSES?
  31. 31. CPU Scheduler needs to account for container CPU requirements and host available CPUs
  32. 32. WHAT DO YOU THINK HAPPENS WHEN? Your container tries to access more than one CPU Your container goes over CPU limits
  33. 33. Totally different from memory Mesos/Kubernetes CPU translates into Docker --cpu- shares
  34. 34. OTHER CONSIDERATIONS
  35. 35. DOCKER AND THE PID 1 ZOMBIE REAPING PROBLEM https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1- zombie-reaping-problem/ Zombie processes are processes that have terminated but have not (yet) been waited for by their parent processes. The init process -- PID 1 -- has a special task. Its task is to "adopt" orphaned child processes
  36. 36. PROCESS ADOPTION
  37. 37. THIS IS A PROBLEM IN DOCKER Jenkins slaves run multiple processes But Jenkins masters too, and they are long running
  38. 38. TINI Systemd or SysV init is too heavyweight for containers All Tini does is spawn a single child (Tini is meant to be run in a container), and wait for it to exit all the while reaping zombies and performing signal forwarding.
  39. 39. PROCESS REAPING Docker 1.9 gave us trouble at scale, rolled back to 1.8 Lots of defunct processes
  40. 40. NETWORKING Multiple services running in the same ports Must redirect from random ports in the host Services running in one host need to access services in other hosts
  41. 41. NETWORKING: SERVICE DISCOVERY DNS is not great, caching can happen at multiple levels marathon-lbuses haproxyand Marathon API A typical nginxreverse proxy is also easy to setup There are more complete solutions like Consul
  42. 42. NETWORKING: SECURITY Prevent/Allow from to container host container container container another host container container in another host
  43. 43. NETWORKING: SECURITY Prevent/Allow from to container host iptables container container --icc=false+ --link, docker0bridge device tricks container another host --ip-forward=false, iptables container container in another host iptables
  44. 44. NETWORKING: SOFTWARE DEFINED NETWORKS Create new custom networks on top of physical networks Allow grouping containers in subnets Not trivial to setup
  45. 45. NETWORKING: SOFTWARE DEFINED NETWORKS Battlefield: Calico, Flannel, Weave and Docker Overlay Network
  46. 46. DOCKER OVERLAY Docker networking with default overlaydriver, using VxLAN # On the Swarm master docker network create --driver overlay --subnet=10.0.9.0/24 my-net Uses Consul, etcd or ZooKeeper as key-value stores
  47. 47. WEAVE UDP and VxLAN backends
  48. 48. COREOS FLANNEL UDP and VxLAN backends Uses etcdfor key-value store
  49. 49. PROJECT CALICO A pure Layer 3 model
  50. 50. SCALING New and interesting problems
  51. 51. TERRAFORM AWS Instances Keypairs Security Groups S3 buckets ELB VPCs
  52. 52. AWS Resource limits: VPCs, S3 snapshots, some instance sizes Rate limits: affect the whole account Retrying is your friend, but with exponential backoff
  53. 53. TERRAFORM OPENSTACK Instances Keypairs Security Groups
  54. 54. OPENSTACK Custom flavors Custom images Different CLI commands There are not two OpenStack installations that are the same
  55. 55. UPGRADES / MAINTENANCE Moving containers from hosts Draining hosts Rolling updates Blue/Green deployment Immutable infrastructure
  56. 56. THANKS csanchez.org csanchez carlossg

×