Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Cassandra and Docker
Two buzzwords or a match made in heaven?
instaclustr.com
@Instaclustr
Who am I and what do I do?
• Ben Bromhead
• Co-founder and CTO of Instaclustr -> www.instaclustr.com
<sales>
• Instaclustr...
Objectives
• A quick intro on docker.
• Why docker matters and how it works.
• Working with Cassandra and docker.
• Runnin...
The Challenge
Static website
Web frontend
DB
Queue
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + boo...
Static website
Web frontend
Background workers
DB
Analytics
Queue
Developmen
t VM
QA Server
Single Prod
Server
Onsite
Clus...
Pre 1960’s transportMul$plicity*of*Goods*
Mul$pilicity*of*
methods*for*
transpor$ng/storing*
Do*I*worry*about*
how*goods*i...
Also a dependency mess
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Solution: Intermodal Shipping containers
Mul$plicity*of*Goods*
Mul$plicity*of*
methods*for*
transpor$ng/storing*
Do*I*worr...
Docker, shipping containers for code
Static website Web frontendUser DB Queue Analytics DB
Development
VM
QA server Public...
Solves the deployment dependency matrix
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developm...
Why docker matters
• Finally Developers have a solution to build once and deploy
anywhere
• Finally Ops/Admin has a soluti...
Docker, how it works.
• Runs anywhere (Linux kernel 2.6.32+)
• Uses lightweight VMs:
• Own process space (namespace)
• Pro...
Docker, how it works.
• Difference between a container and a VM
Virtual Machine Container
Docker, how it works.
• What about the packaging component?
• Uses Union filesystem to create a git like workflow around you...
Docker is it production ready?
Cassandra and Docker
• So how do we get on board the hype train? Without killing
performance or stability?
• Build Cassand...
Docker + Networking
• 1st attempt, throughput dropped in half!
• Writes sucked, streaming sucked, what was going on?
• Qui...
Docker + Networking
• Docker uses Linux Ethernet Bridges for basic software defined
routing. This will hose your network th...
Docker + Filesystem
• Don’t want to throw it out when you upgrade/stop container.
• Use volume mount folders to the underl...
Docker + Filesystem
• The filesystems (AUFS, BTRFS etc) that bring great benefits to
Dockers workflow around building and sna...
Docker + Process Capabilities
• Mlockall permission denied? A process needs CAP_IPC_LOCK &
RLIMIT_MEMLOCK in order to perf...
Docker + SIGTERM propagation
• When stopping the process docker will send a SIGTERM.
• PID 1 does not have default signal ...
Docker + SIGTERM propagation
• Java to the rescue!
• Make sure you run the cassandra bash script with -f (foreground)
• ex...
Docker + SIGTERM propagation
• Tools like OpsCenter Server will have trouble with this.
• Can be fixed using a wacky combin...
Docker + CoreOS
• Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis
etc
• We chose CoreOS (Appeared to be sane, ...
Docker + CoreOS
• Disable automatic updates + restarts (seriously do this)
• Fix logging, otherwise you will log to 3 loca...
Docker + Dev Env
• Docker relies on Linux kernel capabilites… so no native docker in
OS X
• We use OSX for dev, so we run ...
Docker + C* + Dev Env
• How do I run lots of C* instances on a VM or my dev laptop without
it falling over?
• Make it run ...
Docker + C* + Dev Env
• Set Memory to be super low, edit your cassandra-env.sh:
MAX_HEAP_SIZE="128M"	
  
HEAP_NEWSIZE="24M"
Docker + C* + Dev Env
• Tune compaction to have free reign and to smash the disk
concurrent_compactors:	
  1	
  
in_memory...
Docker + C* + Dev Env
• Let’s use HSHA thrift server as it reduces the memory per thread
used.
rpc_server_type:	
  hsha
Docker + C* + Dev Env
• The HSHA server also lets us limit the number of threads serving in
flight requests, but still have...
Docker + C* + Dev Env
• This is Dev! Caches have no power here!
key_cache_size_in_mb:	
  0	
  
reduce_cache_sizes_at:	
  0...
Docker + C* + Dev Env
• How well does this work?!?!
• Will survive running the insane workload in the c* 2.1 new stresstes...
Go forth and conquer!
Questions?
Próxima SlideShare
Cargando en…5
×

Cassandra and Docker Lessons Learned

36.406 visualizaciones

Publicado el

In this talk Ben will walk you through running Cassandra in a docker environment to give you a flexible development environment that uses only a very small set of resources, both locally and with your favorite cloud provider. Lessons learned running Cassandra with a very small set of resources are applicable to both your local development environment and larger, less constrained production deployments.

Publicado en: Ingeniería

Cassandra and Docker Lessons Learned

  1. 1. Cassandra and Docker Two buzzwords or a match made in heaven? instaclustr.com @Instaclustr
  2. 2. Who am I and what do I do? • Ben Bromhead • Co-founder and CTO of Instaclustr -> www.instaclustr.com <sales> • Instaclustr provides Cassandra-as-a-Service in the cloud. • Currently in AWS, Azure and Google Cloud in private beta with more to come. • We currently manage 50+ nodes for various customers, who do various things with it. </sales>
  3. 3. Objectives • A quick intro on docker. • Why docker matters and how it works. • Working with Cassandra and docker. • Running C* in a constrained env w/ docker
  4. 4. The Challenge Static website Web frontend DB Queue Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 Java + Cassandra + Spark Ruby + Rails + sass + Unicorn Redis + redis-sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client DevelopmentVM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers Mul$plicity*of*Stacks* Mul$plicity*of* hardware* environments* Production Cluster Customer Data Center Do*services*and*apps* interact* appropriately?* Can*I*migrate* smoothly*and* quickly?*
  5. 5. Static website Web frontend Background workers DB Analytics Queue Developmen t VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’ s laptop Customer Servers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Dependency madness
  6. 6. Pre 1960’s transportMul$plicity*of*Goods* Mul$pilicity*of* methods*for* transpor$ng/storing* Do*I*worry*about* how*goods*interact* (e.g.*coffee*beans* next*to*spices)* Can*I*transport*quickly* and*smoothly* (e.g.*from*boat*to* train*to*truck)*
  7. 7. Also a dependency mess ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  8. 8. Solution: Intermodal Shipping containers Mul$plicity*of*Goods* Mul$plicity*of* methods*for* transpor$ng/storing* Do*I*worry*about* how*goods*interact* (e.g.*coffee*beans* next*to*spices)* Can*I*transport* quickly*and*smoothly* (e.g.*from*boat*to* train*to*truck)* …in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another! A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.!
  9. 9. Docker, shipping containers for code Static website Web frontendUser DB Queue Analytics DB Development VM QA server Public Cloud Contributor’s laptop Mul$plicity*of*Stacks* Mul$plicity*of* hardware* environments* Production Cluster Customer Data Center Do*services*and*apps* interact* appropriately?* Can*I*migrate* smoothly*and*quickly* …that can be manipulated using standard operations and run consistently on virtually any hardware platform ! An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…!
  10. 10. Solves the deployment dependency matrix Static website Web frontend Background workers User DB Analytics DB Queue Developmen t VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’ s laptop Customer Servers
  11. 11. Why docker matters • Finally Developers have a solution to build once and deploy anywhere • Finally Ops/Admin has a solution to configure anywhere • Finally DevOps is easy • Dev == Test == Staging == Production • Move with speed
  12. 12. Docker, how it works. • Runs anywhere (Linux kernel 2.6.32+) • Uses lightweight VMs: • Own process space (namespace) • Process isolation and resource control (cgroups) • Own network adapter • Own filesystem (chroot) • Linux Analog to Solaris Zones, *BSD jails
  13. 13. Docker, how it works. • Difference between a container and a VM Virtual Machine Container
  14. 14. Docker, how it works. • What about the packaging component? • Uses Union filesystem to create a git like workflow around your deployed code: ! ! Docker! Container! Image! Registry! Push% ! ! ! ! Bins/! Libs! ! ! ! ! App! A! App!Δ!! ! ! ! ! Bins/! Docker'Engine' Docker'Engine' Update' Host'is'now'running'A’’' ' App'Δ'' ' ' ' ' Bins/' ' ' ' ' Bins/' Libs' ' ' ' ' App' A' ' ' ' ' Bins/' ' ' ' ' Bins/' Libs' ' ' ' ' App' A’’' Host'running'A'wants'to'upgrade'to'A’’.' Requests'update.'Gets'only'diffs' '
  15. 15. Docker is it production ready?
  16. 16. Cassandra and Docker • So how do we get on board the hype train? Without killing performance or stability? • Build Cassandra in a docker container, run it, then test. • Run in dev to get comfortable with it. • Talk to others who use it in production • https://github.com/docker/docker/issues - You will spend a lot of time here
  17. 17. Docker + Networking • 1st attempt, throughput dropped in half! • Writes sucked, streaming sucked, what was going on? • Quick check with iperf showed a 50% hit in throughput • Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput. • Use the host network stack instead (—net=host), only saw a ~10% hit on network performance
  18. 18. Docker + Networking • Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput. • Use the host network stack instead (—net=host), only saw a ~10% hit on network performance • Also solves NAT issues in an AWS like networking environment.
  19. 19. Docker + Filesystem • Don’t want to throw it out when you upgrade/stop container. • Use volume mount folders to the underlying host!
  20. 20. Docker + Filesystem • The filesystems (AUFS, BTRFS etc) that bring great benefits to Dockers workflow around building and snapshoting containers are not very good for databases. • UnionFS (AUFS) is terrible for writing lots of big files. • BTRFS is a pain to use from an ops point of view. • Hooray volume mounts use the underlying filesystem.
  21. 21. Docker + Process Capabilities • Mlockall permission denied? A process needs CAP_IPC_LOCK & RLIMIT_MEMLOCK in order to perform this operation. By default docker doesn't assign this to a running container… • Can’t use native memory. Cassandra becomes slooow. • Can use --privileged and be done with it. Kind of lazy though • Use --cap-add instead
  22. 22. Docker + SIGTERM propagation • When stopping the process docker will send a SIGTERM. • PID 1 does not have default signal handlers! • Bad if you use a bash script to launch Cassandra Images shameless copied from https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
  23. 23. Docker + SIGTERM propagation • Java to the rescue! • Make sure you run the cassandra bash script with -f (foreground) • exec causes the JVM to replace the bash process… making the world a happier place
  24. 24. Docker + SIGTERM propagation • Tools like OpsCenter Server will have trouble with this. • Can be fixed using a wacky combination of trap and wait stanzas in your OpsCenter Server script (see http://veithen.github.io/ 2014/11/16/sigterm-propagation.html) • But now you have a bash script that duplicates init/systemd/ supervisord • The debate rages on…
  25. 25. Docker + CoreOS • Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis etc • We chose CoreOS (Appeared to be sane, etcd is cool, systemd if you are into that kind of thing)
  26. 26. Docker + CoreOS • Disable automatic updates + restarts (seriously do this) • Fix logging, otherwise you will log to 3 locations (/var/log/ cassandra, journalctl and dockers json based log • JVM will exit with error 143 (128 + 15 for SIGTERM). Need to ignore that in your systemd service definition.
  27. 27. Docker + Dev Env • Docker relies on Linux kernel capabilites… so no native docker in OS X • We use OSX for dev, so we run vagrant and the CoreOS vagrant file https://github.com/coreos/coreos-vagrant • Look at https://github.com/tobert/cassandra-docker for something more off the shelf
  28. 28. Docker + C* + Dev Env • How do I run lots of C* instances on a VM or my dev laptop without it falling over? • Make it run as slowly, but as stable as possible! • This is actually a great learning exercise as you discover a lot about how Cassandra works under the hood.
  29. 29. Docker + C* + Dev Env • Set Memory to be super low, edit your cassandra-env.sh: MAX_HEAP_SIZE="128M"   HEAP_NEWSIZE="24M"
  30. 30. Docker + C* + Dev Env • Tune compaction to have free reign and to smash the disk concurrent_compactors:  1   in_memory_compaction_limit_in_mb:  2   compaction_throughput_mb_per_sec:  0
  31. 31. Docker + C* + Dev Env • Let’s use HSHA thrift server as it reduces the memory per thread used. rpc_server_type:  hsha
  32. 32. Docker + C* + Dev Env • The HSHA server also lets us limit the number of threads serving in flight requests, but still have a large number of clients connected. concurrent_reads:  4   concurrent_writes:  4   rpc_min_threads:  2   rpc_max_threads:  2 • You can play with these to get the right numbers based on how your clients connect, but keep them low.
  33. 33. Docker + C* + Dev Env • This is Dev! Caches have no power here! key_cache_size_in_mb:  0   reduce_cache_sizes_at:  0   reduce_cache_capacity_to:  0
  34. 34. Docker + C* + Dev Env • How well does this work?!?! • Will survive running the insane workload in the c* 2.1 new stresstest tool. • We run this on AWS t1.micro instances • Sign up at https://www.instaclustr.com and give our new Developer nodes a spin!
  35. 35. Go forth and conquer! Questions?

×