SlideShare una empresa de Scribd logo
1 de 21
Descargar para leer sin conexión
Chaos Testing for Docker Containers
Who am I?
‣Alexei Ledenev (@alexeiled)
‣Chief of Research @codefresh.io
‣Open Source Projects
‣github.com/alexei-led/pumba
‣github.com/codefresh-io/microci
‣#docker #k8s #aws #gcloud
Complex Systems
"Sooner or later, any complex system will fail, and software systems are no exception.
Failure can occur anytime and almost anywhere. So you should never get too comfortable."
Last Year Outages
• IBM Cloud, January 26

• GitLab, January 31

• AWS, February 28

• Microsoft Azure, March 16

• ...

• Visit http://outage.report/
What can we do
to achieve better Quality?
More testing? Better monitoring?
Functional Testing
Performance Testing
Integration Testing
Penetration Testing
Acceptance Testing Log Analytics
Monitoring Alerts
Failure Predictions
Building distributed software today is easier than ever
CAP Theorem
“Of three properties of
shared-data systems
(Consistency, Availability
and tolerance to network
Partitions) only two can be
achieved at any given
moment in time.”
Eric Brewer
Chaos Engineering
• Embrace the failure!
• Defines an empirical approach to resilience testing of distributed software systems 

• Chaos Experiment

- define a "normal/steady" state of the system (e.g. by monitoring a set of system and business
metrics)

- pseudo-randomly inject faults (e.g. by terminating VMs, killing containers or changing network
behavior)

- try to discover system weaknesses by deviation from expected or steady-state behavior 

The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.
 
http://principlesofchaos.org/
https://github.com/Netflix/SimianArmy
Google :// Chaos Monkey for DockerWarthog
What is Pumba(a)?
1. Pumbaa is a well-known supporting character
(warthog) from Disney’s animated film The Lion King

2.  In Swahili, pumbaa means “to be foolish, silly, weak-
minded, careless, negligent”
3. It's also an open source Chaos Testing tool for Docker
containers 

1. https://github.com/gaia-adm/pumba

2. Linux, Windows, MacOS, Docker
What Pumba can do?
• Pumba disturbs Docker runtime environment, injecting different failures 

• The "victim" container can be specified, providing name/s or regex

• Radom selection is also supported (with `--random` flag)

• It's possible to define a repeatable time interval and duration parameters
to better control the Chaos

• Pumba can disturb either single Docker host, Swarm cluster, and
Kubernetes cluster
Pumba Docker Chaos Commands
1. stop running Docker container

2. kill (send termination or other signal) to the main process within a
Docker container

3. remove "victim" containers, with their links and volumes

4. pause all processes within a "victim" Docker container for a
specified time
demo time ...
Examples
# stop random container once in a 10 minutes
$ pumba --random --interval 10m kill --signal SIGSTOP
# every 15 minutes kill `mysql` container and
# every hour remove containers starting with "cf"
$ pumba --interval 15m kill --signal SIGTERM mysql &
$ pumba --interval 1h rm re2:^cf &
# every 5 min randomly kill "worker1" or "worker2" containers
# and every 3 minutes pause "queue" container for 15s
$ pumba --random --interval 5m kill --signal SIGKILL worker1 worker2 &
$ pumba --interval 3m pause --duration 15s queue &
Pumba Network Chaos Commands
1. Pumba can emulate network failures at container level (filter by IP too)

2. delay egress traffic for the specified containers

3. add packet-loss based on different probability loss models (2-3-4 state
Markov, Gilbert, Simple Gilbert and Bernoulli)

4. rate limit egress traffic for the specified containers
# add 3 seconds delay for all outgoing packets
# on (default) network device of Docker container for 5 minutes
$ pumba netem --duration 5m delay --time 3000 mydb
# add a delay of 3000ms ± 30ms,
# with the next random element depending 20% on the last one,
# for all outgoing packets on device of all Docker container,
# with name start with for 10 minutes
$ pumba netem --duration 5m --interface eth1 delay 
--time 3000 --jitter 30 --correlation 20 re2:^hp
# add a delay of 3000ms ± 40ms, where variation in delay
# is described by normal distribution,
# for all outgoing packets on main network device of randomly
# chosen Docker container
# from the specified list, for 5 minutes
$ pumba --random netem --duration 5m delay --time 3000 
--jitter 40 --distribution normal 
container1 container2 container3
Pumba Netem under the hood
• The Linux kernel offers a native framework for routing, bridging, firewalling, address
translation and much else.

• Before a packet leaves the output interface, it passes through Linux Traffic Control (tc). This
component is a powerful tool for scheduling, shaping, classifying and prioritizing traffic.

• The basic component of Linux Traffic Control is the queuing discipline (qdisc).  The
simplest implementation of a qdisc is first in first out (FIFO). There are others too.

• The network emulation (netem) project adds queuing disciplines that emulate wide area
network properties such as latency, jitter, loss, duplication, corruption and reordering.
demo time ...
pumba netem loss: https://asciinema.org/a/82430
pumba netem delay: https://asciinema.org/a/82428
Chaos Engineering for Docker

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Introduction to Amazon EKS
Introduction to Amazon EKSIntroduction to Amazon EKS
Introduction to Amazon EKS
 
Chaos engineering
Chaos engineering Chaos engineering
Chaos engineering
 
DevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and ProcessesDevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and Processes
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introduction
 
Introduction to Istio Service Mesh
Introduction to Istio Service MeshIntroduction to Istio Service Mesh
Introduction to Istio Service Mesh
 
gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!gRPC vs REST: let the battle begin!
gRPC vs REST: let the battle begin!
 
CI/CD on AWS
CI/CD on AWSCI/CD on AWS
CI/CD on AWS
 
보안 사고 예방을 위한 주요 AWS 모범 사례 – 신은수, AWS 보안 담당 솔루션즈 아키텍트:: AWS 온라인 이벤트 – 클라우드 보안 특집
보안 사고 예방을 위한 주요 AWS 모범 사례 – 신은수, AWS 보안 담당 솔루션즈 아키텍트:: AWS 온라인 이벤트 – 클라우드 보안 특집보안 사고 예방을 위한 주요 AWS 모범 사례 – 신은수, AWS 보안 담당 솔루션즈 아키텍트:: AWS 온라인 이벤트 – 클라우드 보안 특집
보안 사고 예방을 위한 주요 AWS 모범 사례 – 신은수, AWS 보안 담당 솔루션즈 아키텍트:: AWS 온라인 이벤트 – 클라우드 보안 특집
 
Terraform
TerraformTerraform
Terraform
 
3. 마이크로 서비스 아키텍쳐
3. 마이크로 서비스 아키텍쳐3. 마이크로 서비스 아키텍쳐
3. 마이크로 서비스 아키텍쳐
 
An introduction to terraform
An introduction to terraformAn introduction to terraform
An introduction to terraform
 
Jenkins Overview
Jenkins OverviewJenkins Overview
Jenkins Overview
 
Practical Chaos Engineering
Practical Chaos EngineeringPractical Chaos Engineering
Practical Chaos Engineering
 
AWS 기반 블록체인 (1부) - 블록체인 환경 구성하기 (박혜영 & 유다니엘, AWS 솔루션즈 아키텍트) :: AWS DevDay2018
AWS 기반 블록체인 (1부) - 블록체인 환경 구성하기 (박혜영 & 유다니엘, AWS 솔루션즈 아키텍트) :: AWS DevDay2018AWS 기반 블록체인 (1부) - 블록체인 환경 구성하기 (박혜영 & 유다니엘, AWS 솔루션즈 아키텍트) :: AWS DevDay2018
AWS 기반 블록체인 (1부) - 블록체인 환경 구성하기 (박혜영 & 유다니엘, AWS 솔루션즈 아키텍트) :: AWS DevDay2018
 
DevOps and AWS
DevOps and AWSDevOps and AWS
DevOps and AWS
 
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
 
Building a scalable microservice architecture with envoy, kubernetes and istio
Building a scalable microservice architecture with envoy, kubernetes and istioBuilding a scalable microservice architecture with envoy, kubernetes and istio
Building a scalable microservice architecture with envoy, kubernetes and istio
 
Prometheus design and philosophy
Prometheus design and philosophy   Prometheus design and philosophy
Prometheus design and philosophy
 
ONNX and MLflow
ONNX and MLflowONNX and MLflow
ONNX and MLflow
 
Site (Service) Reliability Engineering
Site (Service) Reliability EngineeringSite (Service) Reliability Engineering
Site (Service) Reliability Engineering
 

Similar a Chaos Engineering for Docker

Automate drupal deployments with linux containers, docker and vagrant
Automate drupal deployments with linux containers, docker and vagrant Automate drupal deployments with linux containers, docker and vagrant
Automate drupal deployments with linux containers, docker and vagrant
Ricardo Amaro
 

Similar a Chaos Engineering for Docker (20)

DevoxxFR 2016 - 3 degrees of MoM
DevoxxFR 2016 - 3 degrees of MoMDevoxxFR 2016 - 3 degrees of MoM
DevoxxFR 2016 - 3 degrees of MoM
 
Container Performance Analysis Brendan Gregg, Netflix
Container Performance Analysis Brendan Gregg, NetflixContainer Performance Analysis Brendan Gregg, Netflix
Container Performance Analysis Brendan Gregg, Netflix
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
Docker Swarm secrets for creating great FIWARE platforms
Docker Swarm secrets for creating great FIWARE platformsDocker Swarm secrets for creating great FIWARE platforms
Docker Swarm secrets for creating great FIWARE platforms
 
FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms
FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE PlatformsFIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms
FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms
 
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
 
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea LuzzardiWhat's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
 
What's New in Docker 1.12 by Mike Goelzer and Andrea Luzzardi
What's New in Docker 1.12 by Mike Goelzer and Andrea LuzzardiWhat's New in Docker 1.12 by Mike Goelzer and Andrea Luzzardi
What's New in Docker 1.12 by Mike Goelzer and Andrea Luzzardi
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 Presentation
 
Demystfying container-networking
Demystfying container-networkingDemystfying container-networking
Demystfying container-networking
 
Unraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production CloudUnraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production Cloud
 
Tokyo OpenStack Summit 2015: Unraveling Docker Security
Tokyo OpenStack Summit 2015: Unraveling Docker SecurityTokyo OpenStack Summit 2015: Unraveling Docker Security
Tokyo OpenStack Summit 2015: Unraveling Docker Security
 
Kubernetes - Starting with 1.2
Kubernetes  - Starting with 1.2Kubernetes  - Starting with 1.2
Kubernetes - Starting with 1.2
 
Parallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-ModeParallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-Mode
 
AsiaBSDCon2023 - Hardening Emulated Devices in OpenBSD’s vmd(8) Hypervisor
AsiaBSDCon2023 - Hardening Emulated Devices in OpenBSD’s vmd(8) HypervisorAsiaBSDCon2023 - Hardening Emulated Devices in OpenBSD’s vmd(8) Hypervisor
AsiaBSDCon2023 - Hardening Emulated Devices in OpenBSD’s vmd(8) Hypervisor
 
FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)
 
JDO 2019: Tips and Tricks from Docker Captain - Łukasz Lach
JDO 2019: Tips and Tricks from Docker Captain - Łukasz LachJDO 2019: Tips and Tricks from Docker Captain - Łukasz Lach
JDO 2019: Tips and Tricks from Docker Captain - Łukasz Lach
 
Docker advance1
Docker advance1Docker advance1
Docker advance1
 
Automate drupal deployments with linux containers, docker and vagrant
Automate drupal deployments with linux containers, docker and vagrant Automate drupal deployments with linux containers, docker and vagrant
Automate drupal deployments with linux containers, docker and vagrant
 

Último

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
masabamasaba
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
masabamasaba
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
 
The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
shinachiaurasa2
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
masabamasaba
 

Último (20)

Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
Generic or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisions
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
Exploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdfExploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdf
 
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park %in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare
 
The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
%in Durban+277-882-255-28 abortion pills for sale in Durban
%in Durban+277-882-255-28 abortion pills for sale in Durban%in Durban+277-882-255-28 abortion pills for sale in Durban
%in Durban+277-882-255-28 abortion pills for sale in Durban
 
SHRMPro HRMS Software Solutions Presentation
SHRMPro HRMS Software Solutions PresentationSHRMPro HRMS Software Solutions Presentation
SHRMPro HRMS Software Solutions Presentation
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
 

Chaos Engineering for Docker

  • 1. Chaos Testing for Docker Containers
  • 2. Who am I? ‣Alexei Ledenev (@alexeiled) ‣Chief of Research @codefresh.io ‣Open Source Projects ‣github.com/alexei-led/pumba ‣github.com/codefresh-io/microci ‣#docker #k8s #aws #gcloud
  • 3. Complex Systems "Sooner or later, any complex system will fail, and software systems are no exception. Failure can occur anytime and almost anywhere. So you should never get too comfortable."
  • 4. Last Year Outages • IBM Cloud, January 26 • GitLab, January 31 • AWS, February 28 • Microsoft Azure, March 16 • ... • Visit http://outage.report/
  • 5.
  • 6. What can we do to achieve better Quality? More testing? Better monitoring? Functional Testing Performance Testing Integration Testing Penetration Testing Acceptance Testing Log Analytics Monitoring Alerts Failure Predictions
  • 7. Building distributed software today is easier than ever
  • 8. CAP Theorem “Of three properties of shared-data systems (Consistency, Availability and tolerance to network Partitions) only two can be achieved at any given moment in time.” Eric Brewer
  • 9. Chaos Engineering • Embrace the failure! • Defines an empirical approach to resilience testing of distributed software systems • Chaos Experiment - define a "normal/steady" state of the system (e.g. by monitoring a set of system and business metrics) - pseudo-randomly inject faults (e.g. by terminating VMs, killing containers or changing network behavior) - try to discover system weaknesses by deviation from expected or steady-state behavior The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.   http://principlesofchaos.org/
  • 11. Google :// Chaos Monkey for DockerWarthog
  • 12. What is Pumba(a)? 1. Pumbaa is a well-known supporting character (warthog) from Disney’s animated film The Lion King 2.  In Swahili, pumbaa means “to be foolish, silly, weak- minded, careless, negligent” 3. It's also an open source Chaos Testing tool for Docker containers 1. https://github.com/gaia-adm/pumba 2. Linux, Windows, MacOS, Docker
  • 13. What Pumba can do? • Pumba disturbs Docker runtime environment, injecting different failures • The "victim" container can be specified, providing name/s or regex • Radom selection is also supported (with `--random` flag) • It's possible to define a repeatable time interval and duration parameters to better control the Chaos • Pumba can disturb either single Docker host, Swarm cluster, and Kubernetes cluster
  • 14. Pumba Docker Chaos Commands 1. stop running Docker container 2. kill (send termination or other signal) to the main process within a Docker container 3. remove "victim" containers, with their links and volumes 4. pause all processes within a "victim" Docker container for a specified time
  • 16. Examples # stop random container once in a 10 minutes $ pumba --random --interval 10m kill --signal SIGSTOP # every 15 minutes kill `mysql` container and # every hour remove containers starting with "cf" $ pumba --interval 15m kill --signal SIGTERM mysql & $ pumba --interval 1h rm re2:^cf & # every 5 min randomly kill "worker1" or "worker2" containers # and every 3 minutes pause "queue" container for 15s $ pumba --random --interval 5m kill --signal SIGKILL worker1 worker2 & $ pumba --interval 3m pause --duration 15s queue &
  • 17. Pumba Network Chaos Commands 1. Pumba can emulate network failures at container level (filter by IP too) 2. delay egress traffic for the specified containers 3. add packet-loss based on different probability loss models (2-3-4 state Markov, Gilbert, Simple Gilbert and Bernoulli) 4. rate limit egress traffic for the specified containers
  • 18. # add 3 seconds delay for all outgoing packets # on (default) network device of Docker container for 5 minutes $ pumba netem --duration 5m delay --time 3000 mydb # add a delay of 3000ms ± 30ms, # with the next random element depending 20% on the last one, # for all outgoing packets on device of all Docker container, # with name start with for 10 minutes $ pumba netem --duration 5m --interface eth1 delay --time 3000 --jitter 30 --correlation 20 re2:^hp # add a delay of 3000ms ± 40ms, where variation in delay # is described by normal distribution, # for all outgoing packets on main network device of randomly # chosen Docker container # from the specified list, for 5 minutes $ pumba --random netem --duration 5m delay --time 3000 --jitter 40 --distribution normal container1 container2 container3
  • 19. Pumba Netem under the hood • The Linux kernel offers a native framework for routing, bridging, firewalling, address translation and much else. • Before a packet leaves the output interface, it passes through Linux Traffic Control (tc). This component is a powerful tool for scheduling, shaping, classifying and prioritizing traffic. • The basic component of Linux Traffic Control is the queuing discipline (qdisc).  The simplest implementation of a qdisc is first in first out (FIFO). There are others too. • The network emulation (netem) project adds queuing disciplines that emulate wide area network properties such as latency, jitter, loss, duplication, corruption and reordering.
  • 20. demo time ... pumba netem loss: https://asciinema.org/a/82430 pumba netem delay: https://asciinema.org/a/82428