Amazon CI-CD Practices for Software Development Teams

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Curtis Rissi
Sr. Solutions Architect, Amazon Web Services
SRV320
Amazon CI/CD Practices for Software
Development Teams

Agenda
Quick overview of Continuous Integration (CI) and Continuous
Delivery (CD)
Why it matters
CI/CD tooling & techniques
Bringing it all together
High-fives all around

What is continuous integration (CI)?

What is continuous integration?
1. Developers regularly check in
code to a shared, central
repository
2. Automated builds run using the
shared code base
3. Tests are run against the code
base giving feedback to the
developers
Develop
BuildTest

What is continuous delivery (CD)?

What is continuous delivery?
Develop
BuildTest
Develop
Build
Test
Deploy
Test
Release

What is continuous delivery?
1. Successful builds are
automatically deployed to
Staging and Pre-Prod
2. Automated and manual
exploratory tests are
completed on each tier
3. Approval needed for release to
Production
Develop
Build
Test
Deploy
Test
Release

Why does CI/CD matter?

• Find bugs earlier
• Fix bugs faster
• Deliver faster
• Deliver more often
• Unblock developers
• Grow skills faster
Quality
5x
Lower change failure rate
Source: 2017 State of DevOps Report (Puppet)

• Fix bugs faster
• Deliver faster
Delivery
440x
Faster from commit to deploy

• Fix bugs faster
• Deliver faster
Delivery
46x
More frequent deployments

• Fix bugs faster
• Deliver faster
Happy teams
44%
More time spent on new
features and code

Continuous Integration Tooling

Continuous Integration tools
Monitoring / Events
Amazon CloudWatch
Software development
AWS CodeCommit
GitHub
Build & test
AWS CodeBuild

Sample application
Simple Calculator Service
10 2x
Submit
Your result is 20

AWS CodeBuild
• Fully managed build service
• Continuous scaling
• Pay as you go
• Extensible

AWS CodeBuild build spec
version: 0.2
phases:
install:
commands:
- npm install
build:
commands:
- npm test
artifacts:
files:
- '**/*'

So how do we do it?

The continuous integration journey
Develop
BuildTest

10 mph 65 mph 150 mph

Nightly
checks
Branch
checks
Pull request
checks

Technique #1: Nightly checks
1. Nightly checks
2. Branch checks
3. Pull request checks
Run a full build and
unit tests every
night to make sure
that application still
compiles and that
tests still pass.

Technique #2: Branch checks
1. Nightly checks
2. Branch checks
unit tests every time
someone pushes a
new change to a
branch in the source
code repo.

Technique #3: Pull request checks
1. Nightly checks
2. Branch checks
unit tests every time
someone creates a
pull request to get
code reviewed by
the team.

Pull request checks

PR checks: AWS CodeCommit
CodeBuild
build
CodeCommit
pull request
Team
members
Propose
Notify Start
CloudWatch
event
NotifyComment

PR checks: GitHub
AWS CodeBuild
build
GitHub
pull request
Team
members
Start
Report
Propose

PR Checks: GitHub Webhooks

PR checks: Speed boosts
1. Integration tests
2. Parallel builds

#TeamChat: Test stack failures
Oh no. The last deployment to our test stack failed.
Dave 2:15 pm
Looks like the latest code causes a bunch of null pointer exceptions.
Tim 2:16 pm
Why didn’t we catch this in code review?
Clare 2:17 pm

Speed Boost: Integration tests
AWS CodeBuild
build
Test
Amazon ElasticCache
cluster

#TeamChat: Slow checks
Can someone review my pull request?
Clare 2:15 pm
Come back in an hour when the PR build finishes.
Tim 2:16 pm

Speed boost: Parallel builds
GitHub
pull request
Team
members
StartPropose
CodeBuild
builds

#TeamChat: CI nirvana
Isn’t continuous integration great?
Clare 9:48 am
SO GREAT
Dave 9:50 am
Yeah, I feel so productive!
Tim 11:01 am

Pull request checks summary
Implement: Run a build while code is still in review
Feedback loop: Time it takes to build the code
Team impact: Broken code doesn’t block the team
Speed boosts: Integration tests; parallel builds

Continuous integration speed boosts
1. Automate the boring stuff
Library upgrades
2. Surface failures
Email, Slack
3. Check faster
Caching, parallel builds
4. Check more
Integration tests

So it builds … now what?

Let’s get it deployed
Develop
Build
Test
Deploy
Test
Release

Continuous delivery tools
Monitoring
Amazon CloudWatch
Software development
Amazon SNS
AWS Lambda
Build & test
AWS CodeBuild
Deployment
AWS CodeDeploy
AWS CodePipeline

Continuous delivery journey
1. Continuous service testing
2. Manage deployment health
3. Segment production
4. Halt promotions

Continuous service testing

Be aware when a service is unavailable
Problem
A service can stop working at any time for reasons inside or
outside of its control
Consequence
Your service may be unavailable without your team’s knowing
about it

Use synthetic traffic to simulate real users
• Test all business critical functionality (UI and APIs)
• Tests must run quickly
• Measure client latencies
• Check for reachability

Synthetic traffic
How synthetic traffic flows
Amazon
CloudWatch
Alarm
CloudWatch
Events (1m)

CloudWatch
Events (1m)
Synthetic traffic
Synthetic traffic flow. Why two metric streams?
CloudWatch
Alarm

Building a synthetic traffic test

Building a synthetic traffic test
• Keep it simple
• Build logic in Lambda (invoke with CloudWatch Events)
• Capture data in CloudWatch metrics

AWS Lambda synthetic traffic blueprint

Scheduling the synthetic traffic test

Building a synthetic traffic test: Code

Building a synthetic traffic test: Alarming

Release and deploy process: Synthetic traffic
DeployToProd
CodeDeploy
Production
Synthetic traffic
CodeDeploy

Managing deployment health

V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments: Success
Production fleet
Elastic Load Balancing

V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments: Fail
Production fleet

Check for deployment failures in production
Problem
There are no automated tests to verify that a service is working
after a new deployment
Consequence
Each production deployment needs to be checked manually

Add safety to rolling deployments
1. Validate each host’s health
2. Ensure that a minimum percentage of the fleet is healthy
3. Roll back if the deployment failed

Configure AWS CodeDeploy

Step 1: Deployment validation – AppSpec.yml

V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2
Step 1: Working tests raises more issues
Production Fleet
Failed Deployment

4 failures – 60% healthy
MHH 70%, 10 hosts:
V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2
Step 2: Use minimum healthy hosts
Production Fleet
1 failure – 90% healthy

Step 2: Use minimum health hosts: AWS CodeDeploy

Step 3: Roll back when a deployment fails
• AWS CodeDeploy: Configured in deployment group

Release and deploy: Deployment health
DeployToProd
AWS CodeDeploy
Production
Synthetic traffic
AWS CodeDeploy

Segment production

Bad changes must not affect all customers
Pipeline Problem
When a critical issue reaches production, all hosts are affected
Consequence
Bad changes impact all customers

Lower deployment risk by segmenting
1. Break production into multiple segments
2. Deploy to a segment
3. Test a segment after a deployment
4. Repeat 2 & 3 until done

Step 1: Break production into multiple segments
Typical segment types
• Region
• Availability Zone
• Subzonal
• Single host (canary)
US-EAST-1
US-EAST-1A US-EAST-1B

V2 V2 V2V2V1 V1V1
Step 1: Typical deployment segmentation
Availability Zone-based
deployment
deployment
deployment
V2 V2V2V1 V1V1 V2 V2V2V1 V1V1
Production fleet
Post-deployment test
Canary
deployment
V1
Region-based deployment

Step 1: Use deployment groups as segments
Create deployment groups per segment using:
• Tags
• Auto Scaling groups

Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
Step 2: Deploy to each segment
1. Deploy to smallest segment
2. Post-deployment tests
3. Deploy to one Availability Zone
4. Post-deployment tests
5. Deploy to remaining Availability
Zones

Step 3: Test each segment
A deployment is valid if:
• The test has gathered enough data to gain confidence
• CloudWatch metrics
• No service alarms have fired
• CloudWatch alarms
• The test has not timed out
• Code

Add segment tests to your pipeline
Extend AWS CodePipeline with:
• Test Actions
• Lambda Invoke Actions
• Custom Actions
• Approval Actions
1-hour timeout
7-day timeout

Use AWS CodePipeline approvals to trigger tests
Source
MyAppSource
CodeCommit
Deploy
DeployToSegment
CodeDeploy
ValidateSegment
Approval
putApprovalResult
Approval
message
DeployToSegment
CodeDeploy
SNS topic

Use SNS to start an automated approval check

Creating a post-deployment test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
registerDeployTest()
Lambda Function
evaluateDeploy()
Amazon
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy AlarmTimeUsage
SNS topic

Post-deployment test – registerDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
Lambda function
evaluateDeploy()
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
SNS topic

registerDeployTest function – (Node.js 4.3)

Post-deployment test – evaluateDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
Lambda function
evaluateDeploy()
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
SNS topic

approveValidation function (Node.js 4.3)

Canary deployments – They’re different
All production hosts
• Participates in serving production traffic
• Configured as a production instance
• Participates in production metrics stream
Canary hosts
• Has its own metrics stream
• Canary validations use the canary metric stream

Release and deploy: Segment production
Synthetic traffic
AWS CodeDeploy
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToProd
CodeDeploy
Production

Halt promotions

EC2 instance
Change 2Change 3
Don’t change the system under test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
DeployToProd
MyApp
CodeDeploy
Deploys
Change 1

Don’t compound problems during an outage
Pipeline problem
The pipeline is unaware of the health of the infrastructure it is
deploying to
Consequence
Production changes, usually deployments, can make it difficult for
an operator to resolve a production event

Build promotion blockers

Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
DeployToProd
MyApp
CodeDeploy
Change 1Change 2
Auto stop deploying to PRD during an event
CloudWatch
Synthetic
traffic
Deploys
Checks
CloudWatch
Events (1m)
Triggers
EmitsDisables
disableTransition() CloudWatch alarm
EC2 instance
SNS

disableTransition function (Lambda Node.js 4.3)

Enable production deployments – AWS CodePipeline

Summary: Halt promotions
• Halt promotions to production when your production
environment has “issues”
• Automate by disabling stage transitions

Release and deploy: Halt promotions
Synthetic traffic
AWS CodeDeploy
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

Release and deploy process: Ending point
DeployToProd
CodeDeploy
Production
AWS CodeDeploy
Synthetic traffic
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
Production

Continuous delivery summary
Goal: Make your pipeline safer
1. Automated testing and notifications
• Keep pipeline unblocked
2. Identify production issues quickly
• Continuous Production Testing
3. Safely deploy changes
• Manage deployment health
• Segment production
4. Automatically decide when to release changes
• Halt promotions

Bringing it all together…
Develop
Build
Test
Deploy
Test
Release

Code is available online
• https://github.com/aws-samples/aws-codebuild-samples
• https://github.com/awslabs/aws-codepipeline-synthetic-tests
• https://github.com/awslabs/aws-codepipeline-block-production

Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

Thank you!

Acknowledgements
• Original 2016 slides written and prepared by Mark Mansour, Senior Manager,
Continuous Delivery, AWS
• This presentation, “DevOps on AWS: Advanced Continuous Delivery
Techniques,” was originally given at re:Invent 2016 on Nov 30, 2016
• 2017 slides updated by Curtis Bray, Manager, AWS CodePipeline for DEV324
presentation at re:Invent 2017
• 2018 slides updated by Curtis Rissi, Sr. Solutions Architect to incorporate
“Continuous Integration Best Practices for Software Development Teams” by
Clare Liguori, AWS Senior Software Engineer to cover the full CI/CD process

Amazon CI-CD Practices for Software Development Teams

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Amazon CI-CD Practices for Software Development Teams

Similar to Amazon CI-CD Practices for Software Development Teams (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Amazon CI-CD Practices for Software Development Teams