Continuous delivery (CD) enables teams to be more agile and quickens the pace of innovation. Too often, however, teams adopt CD without putting the right safety mechanisms in place. In this talk, we discuss opportunities for you to transform your software release process into a safer one. We explore various DevOps best practices, showcasing sample applications and code with AWS CodePipeline and AWS CodeDeploy. We discuss how to set up delivery pipelines with nonproduction testing stages, failure cases, rollbacks, redundancy, canary testing and blue/green deployments, and monitoring. We discuss continuous delivery practices for deploying to Amazon EC2, AWS Lambda, and containers such as Amazon ECS or AWS Fargate.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Advanced Continuous Delivery
Best Practices
Curtis Rissi
Senior Solutions Architect
AWS
D E V 3 1 7
Felipe Almeida
Senior Software Dev Engineer
AWS Developer Tools

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What you’ll learn from the session
• Some of the continuous deployment best practices of Amazon
• How and why you should employ these practices yourself
• How to achieve continuous deployment nirvana
• Leaving positive feedback in the post-session survey feels good

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is continuous delivery?
Source Build
Pre-
Test
Deploy
Post-
Test

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Basic continuous delivery best practices
• Versioned source
• Automated build
• Automated deployments
• Deploy to > one instance
• Unit tests
• Integration tests
• Continuous delivery
• Operations dashboard
Source
Build
Deploy to
integration stack
Integration tests
Deploy to
production

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tools used in this talk
Monitoring
Amazon CloudWatch
Software development
Amazon SNS
AWS Lambda
Continuous delivery
AWS CodeDeploy
Continuous deployment
AWS CodePipeline

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MyApp
CodeCommit
Source
Build
CodeCommit
Build
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
DeployToProd
CodeDeploy
Production
Source
Build
Deploy to
integration stack
Integration tests
Deploy to
production
Model the release process in CodePipeline
Pipeline Run
Action
Stage
Pipeline
Source change
• starts a run; and
• creates an artifact to be used by
other actions
Change 1

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Starting point
MyApp
CodeCommit
Source
Build
Build
Build
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
DeployToProd
CodeDeploy
Production
Want to dive deep into to do this with Containers and Serverless?
Check the session catalog or YouTube for:
DEV309-R - CI/CD for Serverless and Containerized Applications

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
An automated pipeline should be …
• Defined as code
• Checked into a version control repository, such as AWS CodeCommit
• Able to allow for extensibility through other AWS services or third-
party tools
• Able to provide FAST feedback on the success and failure of pipeline
executions

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Opportunities for automation
• Continuous integration processes: Builds, integration tests, UI testing,
and more
• Health checks
• Application tests
• Synthetic user tests and application performance monitoring
• Notifications and alerts
• AWS CloudWatch Alarms and third-party tools, such as Splunk, Datadog
• Amazon SNS, Slack, Pagerduty, and more

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Build and unit tests
1. Trigger pipeline
on source change
2. Build and unit
tests
3. Deploy to
integration
environment
4. Execute UI tests
5. Execute
integration tests
Tests
Source
MyAppSource
CodeCommit
Build
BuildAndUnitTests
CodeBuild
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnChrome
CodeBuild
IntegTest
End2EndTester

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Notify on failed build and test
Change 1
CloudWatch
Events
(Failed Action)
Tests
Source
MyAppSource
CodeCommit
Build
BuildAndUnitTests
CodeBuild
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnChrome
CodeBuild
IntegTest
End2EndTester
Change 2
Lambda Function
NotifySlackOnPipelineActionFailure()

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Builds on top of our foundation of automation
• Purpose built to verify that a service is working
after a new deployment
• Helps to avoid needing to do this manually
Manage deployment health

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments – Success

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments – Fail

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add safety to rolling deployments
1. Validate each host’s health
2. Ensure a minimum percentage of the fleet is healthy
3. Rollback if the deployment failed

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Deployment validation – AppSpec.yml

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2
Step 1: Working tests raises more issues
Failed deployment

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
4 failures – 60% healthy
MHH 70%, 10 hosts:
V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2
Step 2: Use minimum healthy hosts
1 failure – 90% healthy

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Use minimum health hosts - CodeDeploy

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Rollback when a deployment fails
• CodeDeploy: Configured in deployment group

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy: Deployment health
DeployToProd
CodeDeploy
Production
Synthetic traffic

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blue/green deploys
Failed deployment
Blue group Green group
V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blue/green deploys
Successful deployment
Blue group Green group
V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2
Auto Scaling Auto Scaling

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lower deployment risk by segmenting
• Minimize the impact of deployment failures
• Potentially catch issues before real users do
• Enables you to roll back more quickly, with less impact

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Segmentation overview
1. Break production into multiple segments
2. Deploy to a segment
3. Test a segment after a deployment
4. Repeat 2 & 3 until done

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Break production into multiple segments
Typical segment types:
• Region
• Availability Zone
• Sub-Zonal
• Single host (Canary)
US-EAST-1
US-EAST-1A US-EAST-1B

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V2 V2 V2V2V1 V1V1
Step 1: Typical deployment segmentation
Availability Zone based
deployment
Availability Zone based
deployment
Availability Zone based
deployment
V2 V2V2V1 V1V1 V2 V2V2V1 V1V1
Production fleet
Post-deployment test
Canary
deployment
V1
Region based deployment

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Use deployment groups as segments
Create deployment groups per segment using
• Tags
• Auto Scaling groups

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
Step 2: Deploy to each segment
1. Deploy to smallest segment
2. Post-deployment tests
3. Deploy to one availability zone
4. Post-deployment tests
5. Deploy to remaining availability zones
6. Repeat for other regions
Production – Region 2
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Test each segment
A deployment is valid if
• The test has gathered enough data to gain confidence
• CloudWatch metrics
• No service alarms have fired
• CloudWatch alarms
• The test has not timed out
• Code

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add segment tests to your pipeline
Extend CodePipeline with:
• Test Actions
• Lambda Invoke Actions
• Custom Actions
• Approval Actions
1 hour timeout
7 day timeout

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use CodePipeline approvals to trigger tests
Source
MyAppSource
CodeCommit
Deploy
DeployToSegment
CodeDeploy
ValidateSegment
Approval
putApprovalResult
Approval
message
DeployToSegment
CodeDeploy
SNS Topic

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use SNS to start an automated approval check

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Creating a post-deployment test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
events (1m)
Change 1
Prod-us-east-1a
CodeDeploy
AlarmTimeUsage
SNS Topic

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – registerDeployTest
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
events (1m)
Change 1
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Prod-us-east-1a
CodeDeploy
AlarmTimeUsage
SNS Topic

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
registerDeployTest function – (Node.js 4.3)

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
registerDeployTest function – (Node.js 4.3)

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
registerDeployTest function – (Node.js 4.3)

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – evaluateDeployTest
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
events (1m)
Change 1
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Prod-us-east-1a
CodeDeploy
AlarmTimeUsage
SNS Topic

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
approveValidation function (Node.js 4.3)

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Canary deployments – They’re different
All production hosts
• Participates in serving production traffic
• Configured as a production instance
• Participates in production metrics stream
Canary hosts
• Has its own metrics stream
• Canary validations use the canary metric stream

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: Segment production
• Segment production to reduce impact of a bad change
• Minimum segmentation
• Region
• Canary deployment per region
• Larger service segmentation
• Zonal
• Sub-zonal
• Test each segment before moving on

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-region deployments
• New as of November 2018
• Allows you to deploy to multiple regions from a single pipeline
• Enables you to achieve lower latency and greater availability

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-region deployment actions
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeployR1
CodeDeploy
ValidateCanaryR1
Approval
Prod-us-east-1a
CodeDeploy
CanaryDeployR2
CodeDeploy
ValidateCanaryR2
Approval
Prod-us-west-2a
CodeDeploy
CanaryDeployR3
CodeDeploy
ValidateCanaryR3
Approval
Prod-ap-southeast-2a
CodeDeploy
Deploy R2 Deploy R3

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Block non-compliant pipelines
• Introducing changes or even new pipelines can cause serious problems
• Downtime risk, reputation, and financial loss
• Leverage AWS Config to ensure pipeline compliance before allowing
them to deploy to production environments

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adding safety with AWS Config rules
Build AWS Config rules
• These alert when pipelines are not configured up to company best
practices
Building a pipeline which blocks production pushes on non-compliant
pipelines
• Use approvals to pause production deployments
• Lambda to automatically approve when pipeline is compliant

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approval action-based pipeline safety check
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
safetyCheck
Approval
ProductionDeploy
CodeDeploy
Lambda Function
safetyCheck
Change 1
SNS Topic

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Config rules-based pipeline safety check

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Config rules-based pipeline safety check
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
DeployToProd
MyApp
CodeDeploy
CloudWatch
Event (event-
based)
Lambda Function
disablePushtoProduction

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: Safety and compliance checks in your
pipelines
• Use AWS Config rules and CloudWatch events to automatically
remediate non-compliant pipelines
• Add common action to all pipelines
• Provide developers a best practices pipeline to start from with AWS
CloudFormation templates or AWS Service Catalog
• Implement with approval actions in CodePipeline

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
safetyCheck
Approval
Release and deploy: Gates
Synthetic traffic
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What we’ve learned
Goal: Make your pipeline safer …
1. Identify production issues quickly
Continuous production testing
Automate notifications
2. Safely deploy changes
Manage deployment health
Segment production
3. Improve latency and availability
Deploy across regions
4. Automatically decide when to release changes
Block non-compliant pipelines

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Ending point
DeployToProd
CodeDeploy
Production
Synthetic traffic
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
safetyCheck
Approval
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
Production

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Code is available online
• github.com/awslabs/aws-codepipeline-time-windows
• github.com/awslabs/aws-codepipeline-synthetic-tests
• github.com/awslabs/aws-codepipeline-block-production

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related sessions
• DEV303 – Deploying and Managing .NET Pipelines and Microsoft Workloads
• DEV310 – DevOps on AWS: Choosing the Right Software Deployment Technique
• DEV313 – Infrastructure Continuous Deployment Using AWS CloudFormation
• SVR307 – Application Lifecycle Management in a Serverless World

66. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Curtis Rissi
Felipe Almeida

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Author:
Original 2016 slides written and prepared by Mark Mansour, Senior
Manager, Continuous Delivery, AWS.
This presentation, “DevOps on AWS: Advanced Continuous Delivery
Techniques,” was originally given at re:Invent 2016 on Nov 30, 2016.
2017 slides updated by Curtis Bray, Manager, AWS CodePipeline for
DEV324 presentation at re:Invent 2017.
This presentation was updated for Re:Invent 2018 by Leo Zhadanovsky,
Curtis Rissi and Felipe Madrial