SlideShare una empresa de Scribd logo
1 de 24
Descargar para leer sin conexión
CI/CD Pipelines for Microservices
Best Practices
DAN GARFIELD
Dan
Garfield
Chief Technology Evangelist
@todaywasawesome
Agenda
1. Why microservices
2. How Expedia approached
microservice CI/CD
3. How we do it at Codefresh
Why Microservices?
Monolith 1 Monolith 2
Why Microservices?
Multiple services with single UI&UXMultiple monoliths with diff. UI&UX
Expedia corporate travel re-architecture
Unified cars UI
Car sortCar search
Cars
(Egencia)
Cars (ECT)
Cars
(Travelforce)
Booking service
Phone apps.
Moving to Microservices at Expedia
Approach
● Consolidate code bases and
● Build shared libraries for global platform. Ex:
○ Logging service, monitoring service
● Rely on manual integration testing
● Standardize CI/CD pipelines
● Use Maven for modularity
● Migrate to cloud from on-prem
Issues faced
● Geographically distributed
● Tools consolidation was hard
● Too many pipelines as
microservices grew.
( 100 pipelines → 1000+ )
● Pipelines not modular or
re-usable
● Jenkins master-slave issues
● Copypasta causing bad patterns
● Central team could not keep up
● Plugin upgrade was a nightmare
Moving to Microservices at Expedia
Lessons learned and Recommendations
● CI/CD templates should have been prioritized higher than “business needs”
● Bootstrapping new projects should have been externalized from the microservice and
adding a new microservice should have full pipeline setup once a repo is created
● A modular pipeline approach would ease the pain caused by different versions
● Reusability in CI/CD platform is critical
Single pipeline per
project
Can be
complex/difficult to
be maintain
Usually led by a single
team (anti-devops)
Organizing pipelines for monolithic applications
Scalability issues with microservice pipelines
Does this look
like a plan??
Shared libraries are not the solution.
Requires everyone to use
same version of library
Libraries often rely on each
other in complex ways
Changes have to go to
admins
Leads to big stability
problems
Relies on proprietary API
Organizing pipelines for monolithic applications
How Codefresh does
CI/CD for Microservices
1. Container-based pipelines
2. Shared pipelines
3. Deployment testing
CODEFRESH ARCHITECTURE DIAGRAM
MULTI CLOUD
Each task is built into a
Docker image.
Users can self-serve
these images.
Images DO NOT rely
on each other.
Containers can be
anything: go/node/c++
Container-based pipelines
Huge open source library at steps.codefresh.io
Maintain a single
pipeline
Make microservices
uniform
Change behavior
based on context
Use a single pipeline that operates with context
Triggers carry their context
Codebase
Tests
Docker-compose
Helm chart
...
Dependencies
...
Git checkout
Demo time!
Why Canary?
Usefulness of early testing
Complexity of
Infrastructure
Testing early
becomes less useful
as infrastructure
complexity rises
https://codefresh.io/events/canary-deployment-helm-istio-codefresh/
Summary
Shared pipelines > libraries
Reusable Docker images > Copypasta
Deployment validation with canary
Read the blog post at
https://codefresh.io/continuous-deployment/
ci-cd-pipelines-microservices/
Questions?
Dan Garfield
@todaywasawsome
Want to try it yourself?
Open a FREE account
today at
Codefresh.io
Kostis Kapelonis
@codepipes

Más contenido relacionado

Más de Codefresh

Más de Codefresh (20)

Why You Should be Using Multi-stage Docker Builds in 2019
Why You Should be Using Multi-stage Docker Builds in 2019Why You Should be Using Multi-stage Docker Builds in 2019
Why You Should be Using Multi-stage Docker Builds in 2019
 
Deploy Secure Cloud-Native Apps Fast
Deploy Secure Cloud-Native Apps Fast Deploy Secure Cloud-Native Apps Fast
Deploy Secure Cloud-Native Apps Fast
 
CICD Pipelines for Microservices Best Practices
CICD Pipelines for Microservices Best Practices CICD Pipelines for Microservices Best Practices
CICD Pipelines for Microservices Best Practices
 
Codefresh CICD New Features Launch! May 2019
Codefresh CICD New Features Launch! May 2019Codefresh CICD New Features Launch! May 2019
Codefresh CICD New Features Launch! May 2019
 
Terraform GitOps on Codefresh
Terraform GitOps on CodefreshTerraform GitOps on Codefresh
Terraform GitOps on Codefresh
 
Adding Container Image Scanning to Your Codefresh Pipelines with Anchore
Adding Container Image Scanning to Your Codefresh Pipelines with AnchoreAdding Container Image Scanning to Your Codefresh Pipelines with Anchore
Adding Container Image Scanning to Your Codefresh Pipelines with Anchore
 
Image scanning using Clair
Image scanning using Clair Image scanning using Clair
Image scanning using Clair
 
Updating Kubernetes With Helm Charts: Build, Test, Deploy with Codefresh and...
 Updating Kubernetes With Helm Charts: Build, Test, Deploy with Codefresh and... Updating Kubernetes With Helm Charts: Build, Test, Deploy with Codefresh and...
Updating Kubernetes With Helm Charts: Build, Test, Deploy with Codefresh and...
 
Docker based-Pipelines with Codefresh
Docker based-Pipelines with CodefreshDocker based-Pipelines with Codefresh
Docker based-Pipelines with Codefresh
 
Automated Serverless Pipelines with #GitOps on Codefresh
Automated Serverless Pipelines with #GitOps on CodefreshAutomated Serverless Pipelines with #GitOps on Codefresh
Automated Serverless Pipelines with #GitOps on Codefresh
 
Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...
Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...
Discovering and Fixing Dependency Vulnerabilities for Kubernetes apps with Sn...
 
Net Pipeline on Windows Kubernetes
Net Pipeline on Windows KubernetesNet Pipeline on Windows Kubernetes
Net Pipeline on Windows Kubernetes
 
Multi-cloud CI/CD with failover powered by K8s, Istio, Helm, and Codefresh
Multi-cloud CI/CD with failover powered by K8s, Istio, Helm, and Codefresh Multi-cloud CI/CD with failover powered by K8s, Istio, Helm, and Codefresh
Multi-cloud CI/CD with failover powered by K8s, Istio, Helm, and Codefresh
 
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProSkip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a Pro
 
Istio + Helm + Canary Webinar
Istio + Helm + Canary WebinarIstio + Helm + Canary Webinar
Istio + Helm + Canary Webinar
 
Selenium Testing your Kubernetes Apps with Machine Learning and Testim
Selenium Testing your Kubernetes Apps with Machine Learning and TestimSelenium Testing your Kubernetes Apps with Machine Learning and Testim
Selenium Testing your Kubernetes Apps with Machine Learning and Testim
 
Continuous Delivery for Kubernetes Apps with Helm and ChartMuseum
Continuous Delivery for Kubernetes Apps with Helm and ChartMuseumContinuous Delivery for Kubernetes Apps with Helm and ChartMuseum
Continuous Delivery for Kubernetes Apps with Helm and ChartMuseum
 
New Features Webinar-April
New Features Webinar-AprilNew Features Webinar-April
New Features Webinar-April
 
Continuous Delivery of Stateful Applications with Kubernetes (in Production)
Continuous Delivery of Stateful Applications with Kubernetes (in Production)Continuous Delivery of Stateful Applications with Kubernetes (in Production)
Continuous Delivery of Stateful Applications with Kubernetes (in Production)
 
5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes
 

Último

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Último (20)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

CICD Pipelines for Microservices Best Practices

  • 1. CI/CD Pipelines for Microservices Best Practices DAN GARFIELD
  • 3. Agenda 1. Why microservices 2. How Expedia approached microservice CI/CD 3. How we do it at Codefresh
  • 5.
  • 8. Multiple services with single UI&UXMultiple monoliths with diff. UI&UX Expedia corporate travel re-architecture Unified cars UI Car sortCar search Cars (Egencia) Cars (ECT) Cars (Travelforce) Booking service Phone apps.
  • 9. Moving to Microservices at Expedia Approach ● Consolidate code bases and ● Build shared libraries for global platform. Ex: ○ Logging service, monitoring service ● Rely on manual integration testing ● Standardize CI/CD pipelines ● Use Maven for modularity ● Migrate to cloud from on-prem Issues faced ● Geographically distributed ● Tools consolidation was hard ● Too many pipelines as microservices grew. ( 100 pipelines → 1000+ ) ● Pipelines not modular or re-usable ● Jenkins master-slave issues ● Copypasta causing bad patterns ● Central team could not keep up ● Plugin upgrade was a nightmare
  • 10. Moving to Microservices at Expedia Lessons learned and Recommendations ● CI/CD templates should have been prioritized higher than “business needs” ● Bootstrapping new projects should have been externalized from the microservice and adding a new microservice should have full pipeline setup once a repo is created ● A modular pipeline approach would ease the pain caused by different versions ● Reusability in CI/CD platform is critical
  • 11. Single pipeline per project Can be complex/difficult to be maintain Usually led by a single team (anti-devops) Organizing pipelines for monolithic applications
  • 12. Scalability issues with microservice pipelines Does this look like a plan??
  • 13. Shared libraries are not the solution.
  • 14. Requires everyone to use same version of library Libraries often rely on each other in complex ways Changes have to go to admins Leads to big stability problems Relies on proprietary API Organizing pipelines for monolithic applications
  • 15. How Codefresh does CI/CD for Microservices 1. Container-based pipelines 2. Shared pipelines 3. Deployment testing
  • 17. Each task is built into a Docker image. Users can self-serve these images. Images DO NOT rely on each other. Containers can be anything: go/node/c++ Container-based pipelines
  • 18. Huge open source library at steps.codefresh.io
  • 19. Maintain a single pipeline Make microservices uniform Change behavior based on context Use a single pipeline that operates with context
  • 20. Triggers carry their context Codebase Tests Docker-compose Helm chart ... Dependencies ... Git checkout
  • 22. Why Canary? Usefulness of early testing Complexity of Infrastructure Testing early becomes less useful as infrastructure complexity rises https://codefresh.io/events/canary-deployment-helm-istio-codefresh/
  • 23. Summary Shared pipelines > libraries Reusable Docker images > Copypasta Deployment validation with canary Read the blog post at https://codefresh.io/continuous-deployment/ ci-cd-pipelines-microservices/
  • 24. Questions? Dan Garfield @todaywasawsome Want to try it yourself? Open a FREE account today at Codefresh.io Kostis Kapelonis @codepipes