DevOps and AWS

1. DevOps and AWS
Shiva Narayanaswamy
Solution Architect

2. ~11.6s
Mean time between
deployments
(weekday)
~1,079
Max number of
deployments in a
single hour
~10,000
Mean number of hosts
simultaneously
receiving a
deployment
~30,000
Max number of hosts
simultaneously
receiving a
deployment
DEPLOYMENTS AT AMAZON.COM

3. 75%
Reduction in
outages triggered
by software
deployments
since 2006
90%
Reduction in
outage minutes
triggered by
software
deployments
Business Value of Frequent Deployments
~0.001%
Software
deployments
cause an
outage

4. DevOps Culture

5. Innovation & Stability
Business
Development Operations
Build it faster Keep it stable

6. Continuous Innovation
Business
Idea
Implementation
Testing
Release
Customer
Feedback
Customer

7. Deploying More Frequently Lowers Risk
Smaller Effort
“Minimized Risk”
Frequent Release Events:
“Agile Methodology”
Time
Change
Rare Release Events:
“Waterfall Methodology”
Larger Effort
“Increased Risk”
Time
Change

8. Knowledge is power
Requirements
Gathering
Release
Development
and Check In
Test and QA
Some Learning
Lots of Learning
Minimal Learning
Cycle Time

9. Change is inevitable
• Change is the root cause of most outages
• This leaves two approaches:
-­ Make it a big deal
-­ Small iterative non events

10. “Do painful things more
frequently, so you can
make it less painful…”
-­ Adrian Cockcroft, Architect, Netflix

11. Waterfall
Analyze Design Build Test Deploy
Agile
Sprint
Deploy
Agile-­er
A D B T
Sprint
A D B T D
Sprint Sprint
A D B T D A D B T D
Agile-­er
Dev QA Ops
Dev / QA Ops
Dev/QA/Ops Dev/QA/Ops Dev/QA/Ops
Sprint
A D B T
Sprint
A D B T

12. DevOps is a Re-Org!
-­ Adrian Cockcroft, Architect, Netflix
RE

13. – A philosophy? Cultural change? Paradigm shift ?
– Alignment of development and IT operations with better communication and
collaboration ?
– Improvement in software deployment ?
– Breaking down the barriers between development and IT operations ?
– Akin to Agile software development applied to infrastructure and IT
operations
– Set of tools and processes
– It’s all of the above!
Code
W
a
l
l
Developer IT Operations
DevOps

14. Business Case Requirements Use Case Features Plan Go to market
Business
Design Code Refactor Unit Test Bug Fix Deliver
Developers
(application)
Provision Configure Orchestrate Deploy Report Monitor
IT Operations
Agile
Development
DevOps
• Continuous Integration
• Continuous Deployment
• IT Automation
• Application Management
• Iterative development
• Scrum, sprints, stories
• Velocity
Business
Agility
IT
Agility
Business and IT agility

15. Building Teams
• Encourage a fail fast, learn quick mindset
• Foster innovation and accountability
• Promote open and honest sharing of lessons
learned
• Build trust across organizational boundaries
-­ Include all the stakeholders
• Expect that it will be a work-­in-­progress

16. 1. Customer Obsession
2. Ownership
3. Invent and Simplify
4. Are Right, A Lot
5. Hire and Develop the Best
6. Insist on the Highest
Standards
7. Think Big
8. Bias for Action
9. Frugality
10. Learn and Be Curious
11. Earn Trust of Others
12. Dive Deep
13. Have Backbone;; Disagree
and Commit
14. Deliver Results
Amazon Leadership Principles

17. DevOps Technology

18. The DevOps Stack
Continuous Deployment
Delivery Pipelines
Deployment Automation
Continuous
Integration
Automated
Testing
Configuration
Management
Agile
Communication

19. • Infrastructure as code
• Application and Infrastructure version management
• Continuous Integration
• Test Automation
• Continuous Deployment
• Monitoring and logging
DevOps Practices

20. Version
Control
CI Server
Package
Builder
Deploy
ServerCommit to
Git/master
Dev
Get /
Pull
Code
AMIs
Distributed Builds
Run Tests in parallel
Staging Env
Test Env
Code
Config
Tests
Prod Env
Push
Config
Install
Create
Repo
CloudFormation
Templates for Env
Generate
Sample Application Cycle

21. Version
Control
CI Server
Package
Builder
Deploy
ServerCommit to
Git/master
Dev
Get /
Pull
Code
AMIs
Send Build Report to Dev
Stop everything if build failed
Distributed Builds
Run Tests in parallel
Staging Env
Test Env
Code
Config
Tests
Prod Env
Push
Config
Install
Create
Repo
CloudFormation
Templates for Env
Generate
Continuous Integration

22. Continuous Integration
• Every developer check-­in initiates a build
-­ Builds are fully automated
-­ Tests validate every check-­in
• Instant feedback for developers
-­ The build must remain fast
• Minimizes the impact of broken builds
-­ Bugs don’t accumulate
• Drives the culture of small, frequent releases

23. Version
Control
CI Server
Package
Builder
Deploy
ServerCommit to
Git/master
Dev
Get /
Pull
Code
AMIs
Send Build Report to Dev
Stop everything if build failed
Distributed Builds
Run Tests in parallel
Staging Env
Test Env
Code
Config
Tests
Prod Env
Push
Config
Install
Create
Repo
CloudFormation
Templates for Env
Generate
Continuous Delivery/Deployment

24. Continuous Deployment vs Delivery

25. • Automated, repeatable process to push changes
• Hardens, de-­risks the deployment process
• Immediate feedback from users
• Supports A/B testing or “We test customer
reactions to features in production”
• Gives us a breadth of data points across our
applications
Continuous Delivery

26. Automated Testing
• Robust, automated testing drives confidence in
the entire process
• Testing is not viewed as a “phase” of a project
-­ Testing starts on day one
• View failures as successes
• Don’t overlook testing the infrastructure
-­ Are you ready for the Simian Army?
• Testing is everyone’s responsibility

27. Version Control
Build/
Compile
Code
Dev
Unit Test
App Code
IT Ops
DR Env
Test Env
Prod Env
Dev Env
Application
Write
App
Code
Infrastructure
CloudFormation
tar, war, zip
yum, rpmDeploy
App
Package
Applicatio
n
Deploy application
only
Deploy infrastructure
only
AMI
Build
AMIs
Validate
Template
s
Write
Infra
Code
Deploy
Infras
Automate
Deployme
nt
Artifact Repository
Continuous Integration / Deployment & Automation

28. DevOps Stack on AWS
28
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
CloudWatchCloudFormationCodeDeploy
CodeCommit CodePipeline
AWS Opsworks
AWS Elastic Container Service

29. CodeCommit
• Data redundancy across Availability Zones
• Data-­at-­rest encryption
• Integrated with AWS Identity and Access Management
• No repo size limit
git push CodeCommit
Git objects
in Amazon S3
Git index
in Amazon
DynamoDB
Encryption key
in AWS KMS
SSH or HTTPS
Secure, scalable, and managed Git source control

30. Same Git experience
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
Cloning into 'aws-cli'...
Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (9900/9900), done.
Checking connectivity... done.
$ nano README.rst
$ git commit -am 'updated README'
[master 4fa0318] updated README
1 file changed, 1 insertion(+)
$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote:
To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
4dacd6d..4fa0318 master -> master

31. CodePipeline
• Customizable workflow engine
• Integrate with partner and custom systems
• Visual editor and status
Continuous delivery and release automation, just like Amazon
Build
1) Build
2) Unit test
1) Deploy
2) UI test
Source Beta Production
1) Deploy
2) Load test
Gamma
1) Deploy region1
2) Deploy region2
3) Deploy region3

32. CodePipeline : Mockup

33. CodeDeploy
• Scale from 1 instance to thousands
• Deploy without downtime
• Centralize deployment control and monitoring
Staging
CodeDeployv1, v2, v3
Production
Dev
Coordinate automated deployments, just like Amazon
Application
revisions
Deployment groups

34. Step 1: Package your application (with an AppSpec
file)
version: 0.0
os: linux
files:
- source: chef/
destination: /etc/chef/codedeploy
- source: target/hello.war
destination: /var/lib/tomcat6/webapps
hooks:
ApplicationStop:
- location: deploy_hooks/stop-tomcat.sh
BeforeInstall:
- location: deploy_hooks/install-chef.sh
AfterInstall:
- location: deploy_hooks/librarian-install.sh
ApplicationStart:
- location: deploy_hooks/chef-solo.sh
ValidateService:
- location: deploy_hooks/verify_service.sh

35. Step 1: Package your application (with an AppSpec
file)
version: 0.0
os: linux
files:
- source: chef/
destination: /etc/chef/codedeploy
- source: target/hello.war
destination: /var/lib/tomcat6/webapps
hooks:
ApplicationStop:
- location: deploy_hooks/stop-tomcat.sh
BeforeInstall:
- location: deploy_hooks/install-chef.sh
AfterInstall:
- location: deploy_hooks/librarian-install.sh
ApplicationStart:
- location: deploy_hooks/chef-solo.sh
ValidateService:
- location: deploy_hooks/verify_service.sh

36. Step 1: Package your application (with an AppSpec
file)
version: 0.0
os: linux
files:
- source: chef/
destination: /etc/chef/codedeploy
- source: target/hello.war
destination: /var/lib/tomcat6/webapps
hooks:
ApplicationStop:
- location: deploy_hooks/stop-tomcat.sh
BeforeInstall:
- location: deploy_hooks/install-chef.sh
AfterInstall:
- location: deploy_hooks/librarian-install.sh
ApplicationStart:
- location: deploy_hooks/chef-solo.sh
ValidateService:
- location: deploy_hooks/verify_service.sh

37. Step 2: Set up your target environments
Agent Agent Agent
Staging
Agent Agent
Agent Agent
Agent
Agent
Production
Deployment groupDeployment group
Group instances by:
• Auto Scaling group
• Amazon EC2 tag
• On-­premises tag

38. Step 3: Deploy!
aws deploy create-deployment
--application-name MyApp
--deployment-group-name TargetGroup
--s3-location bucket=MyBucket,key=MyApp.zip
AWS CLI & SDKs
AWS Console
CI / CD Partners
GitHub

39. Deployment config – Choose speed
v2 v1 v1 v1 v1 v1 v1 v1
v2 v2 v2 v2 v1 v1 v1 v1
v2 v2 v2 v2 v2 v2 v2 v2
One-­at-­a-­time
Half-­at-­a-­time
All-­at-­once

40. Rolling update – Deploy without downtime
v1v1 v1
Load Balancer

41. Rolling update – Deploy without downtime
v1v2 v1
Load Balancer

42. Rolling update – Deploy without downtime
v2v2 v1
Load Balancer

43. Rolling update – Deploy without downtime
v2v2v2
Load Balancer

44. Rolling update – Deploy without downtime
v2v2 v2
Load Balancer

45. Health Tracking – Catch deployment problems
v2v2 v2
Load Balancer

46. Health tracking – Catch deployment problems
v3 v2 v2Stop
Load Balancer

47. Health tracking – Catch deployment problems
v2v2 v2
Load Balancer
Rollback

48. Health tracking – Catch deployment problems
v2v2 v2
Load Balancer

49. DevOps Stack on AWS
49
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
CloudWatchCloudFormationCodeDeploy
CodeCommit CodePipeline
AWS Opsworks
AWS Elastic Container Service

50. Automate with AWS CLI

51. Automate with AWS SDK

52. § Infrastructure as Code
§ Integrates with version control
§ JSON format
§ Templates
§ Stacks
§ Supports most AWS resource typesAWS CloudFormation
AWS Cloudformation

53. {
"Description" : "Create an EC2 instance running the Amazon Linux 32 bit AMI.”,
"Parameters" : {
"KeyPair" : {
"Description" : "The EC2 Key Pair to allow SSH access to the instance",
"Type" : "String"
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyPair" },
"ImageId" : "ami-­75g0061f”,
“InstanceType” : “m1.medium”
}
}
},
"Outputs" : {
"InstanceId" : {
"Description" : "The InstanceId of the newly created EC2 instance",
"Value" : { "Ref" : "Ec2Instance” }
}
}
}
AWS Cloudformation

54. Template File
Defining Stack
Git
Subversion
Mercurial
Dev
Test
Prod
The entire application can be
represented in an AWS
CloudFormation template.
Use the version
control system of
your choice to store
and track changes to
this template
Build out multiple
environments, such
as for Development,
Test, and Production
using the template
AWS Cloudformation

55. Describing Infrastructure with Code
Developers
&
Operations
Internal
Git
CI Server
Pre-­commit
Hook
Testing Environment Subnet
CI Workers
Dev Environment VPC Subnet
DEV WEB
ELBDev Stack
Tier 1
Dev Stack
Tier 2
Dev MySQL
DB Instance
DEV APP
ELB
VPN
TUNNEL
VPN facing VPC Subnet
Internet
Gateway
VPN
Endpoint
Dev Admin
Instance
NAT Instance
Amazon S3
Amazon
DynamoDB
Amazon SQS
Amazon
CloudFront
Amazon
Route 53
This entire infrastructure stack can be constructed, configured, and deployed with code:

56. Infrastructure Code
The code that describes infrastructure should inherit
the same values applied to application code
– Not JUST revision control
– Make use of bug tracking/ticketing systems
– Peer reviews of changes before they happen
– Establish infrastructure code patterns/designs
– Test infrastructure changes like code changes

57. • AWS OpsWorks makes it easy to deploy
& operate apps of all shapes and sizes
• Define configurations for your
entire environment in a format that
you can maintain and version
just like your application source code
• Uses the Chef framework so you can
bring your own recipes or leverage
100’s of community-­built configurations
AWS Opsworks

58. • Quickly deploy and manage applications
in the AWS cloud without worrying about
the application infrastructure
• Automatically handles all
the details of resource provisioning,
load balancing & auto-­scaling
• Launch your Java, PHP, .NET, Node.js,
Python, Ruby and Docker applications
in a matter of minutes
AWS Beanstalk

59. WWW
server
WWW
server
WWW
server
WWW
server
Logging
Service
Pay
Service
Process
Service
Process
Service
Process
Service
Process
Service
Cluster of underlying
EC2 instances
WWW containers Application-­specific microservices
WWW
server
WWW
server
Scale out
at the
container
level
Launching
containers is in
the order of
seconds – very
fast to react
AWS Elastic Container Service

60. AWS Tools
• CloudFormation
• Elastic Beanstalk
• OpsWorks
3rd-­Party Tools
• Chef
• Puppet
What are we deploying? How much control do we want?

61. • Deploy in place
– Deploy all at once (Service outage)
– Rolling updates
• Blue-­Green Deployment
– Discrete environment
• Multiple environments from branches
• Support A/B testing
• “Rolling DNS”
• Alternate Blue-­Green (Red-­Black?) deployment
• Alternate auto scaling group
• Avoid messing with DNS
Deployment Approaches

62. CodeDeploy
ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Deploy in place – Rolling update
ELB
S3
EC2

63. ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Deploy in place – Rolling update

64. ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Deploy in place – Rolling update

65. ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Deploy in place – Rolling update

66. ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Deploy in place – Rolling update

67. ALMA Service Icons
Simple Deploy System
Simple Deploy System
Simple Deploy System
Pipelines
Pipelines
Pipelines
Deploy in place – Rolling update

68. Amazon
Route 53
EC2
Instances
ELB
100%
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node
Blue-­Green Deployment

69. Amazon
Route 53
EC2
Instances
EL
B
EC2
Instances
EL
B
100%
UAT
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node
Blue-­Green Deployment

70. Amazon
Route 53
EC2
Instances
EL
B
EC2
Instances
EL
B
90% 10%
DynamoDB MySQL
RDS
Instance
ElastiCach
e Cache
Node
Blue-­Green Deployment

71. Amazon
Route 53
EC2
Instances
ELB
EC2
Instances
ELB
50% 50%
DynamoDB MySQL
RDS
Instance
ElastiCache
Cache
Node
Blue-­Green Deployment

72. Amazon
Route 53
EC2
Instances
ELB
EC2
Instances
ELB
0% 100%
DynamoDB MySQL
RDS
Instance
ElastiCache
Cache Node
Blue-­Green Deployment

73. Amazon
Route 53
EC2
Instances
EL
B
EC2
Instances
ELB
0% 100%
DynamoDB MySQL
RDS
Instance
ElastiCache
Cache
Node
Blue-­Green Deployment

74. Auto
Scaling
Group
V1
ELB
Amazon
Relational
Database
Service (RDS)
Red-­Black Deployment

75. Auto
Scaling
Group
V1
Auto Scaling
Group
V2
ELB
Amazon
Relational
Database
Service (RDS)
UAT
Red-­Black Deployment

76. Auto
Scaling
Group
V1
Auto Scaling
Group
V2
ELB
Amazon
Relational
Database
Service (RDS)
ALM | Red-­Black Deployment

77. Auto Scaling
Group
V2
ELB
Amazon
Relational
Database
Service (RDS)
Red-­Black Deployment

78. Monitoring & Metrics : The DevOps Pulse
• Track every resource possible
• Alert on services, availability, and response times
• Capture, learn, and improve
• Share access with everyone on the team
• Plan metrics and monitoring into the lifecycle

81. Where do I go from here?
• Collect Metrics. Graph anything that moves
• Log everything, Centralize logging, Log
Analytics
• Infrastructure as Code
• Automated configuration management
• One click environment creation
• CI-­CD pipelines
• Automated testing