DevOps, Common use cases, Architectures, Best Practices

1. DevOps
Daniel Zoltak &
Shiva N
Use Cases,Architectures & Best Practices

2. Agenda
• Culture
• Technology
• Your experiences

3. • Breakdown the barriers
• Work as one team end to end
• Support business and IT agility
• Collaboration & Communication
• Treat Infrastructure as code
• Automate everything
• Test everything
• Measure & monitor everything
Agenda…
Culture
Technology

4. Competing Forces
Business
Development Operations
Build it faster Keep it stable

5. Change is inevitable
• Change is the root cause of most outages
• This leaves two approaches:
- Make it a big deal
- Small iterative non events

6. “Do painful things more
frequently, so you can
make it less painful…”
- Adrian Cockcroft, Architect, Netflix

7. Deploying More Frequently Lowers Risk
Smaller Effort
“Minimized Risk”
Frequent Release Events:
“Agile Methodology”
Time
Change
Rare Release Events:
“Waterfall Methodology”
Larger Effort
“Increased Risk”
Time
Change

8. Knowledge is power
Requirements
Gathering
Release
Development
and Check In
Test and QA
Some Learning
Lots of Learning
Minimal Learning
Cycle Time

9. “How long would it take your organization
to deploy a change that involves just one
single line of code?”
“Do you do this on a repeatable, reliable
basis?”
- Mary & Tom Poppendieck, Lean Software Development

10. Waterfall
Analyze
Design
Build
Test
Deploy
Agile
Sprint
Test
Deploy
Agile-er
A
D
B
A
D
B
Sprint
Sprint
A
D
B
Sprint
A
D
B
T
D
Sprint
Sprint
A
D
B
T
D
A
D
B
T
D
Agile-er
Dev
QA
Ops
Dev
QA
Ops
Dev/QA/Ops
Dev/QA/Ops
Dev/QA/Ops

11. What is it?
• A philosophy? Cultural change? Paradigm shift ?
• Alignment of development and IT operations with better communication
and collaboration ?
• Improvement in software deployment ?
• Breaking down the barriers between development and IT operations ?
• Akin to Agile software development applied to infrastructure and IT
operations
• Set of tools and processes
• It’s all of the above!
Code
W
a
l
l
Developer IT Operations
DevOps

12. Building Teams
• Make DevOps everyone’s job
- No silo/walled off DevOps teams
• Encourage developers to participate openly in the
automation of operations code
• Embolden operations participation in testing and
automation of application code
• “Take pride” in how fast and frequent you deploy

13. DevOps Team Anti Patterns
Dev
Ops
Separate Silos
Dev
Ops
DevOps
Separate DevOps Silo
Dev
Ops
DevOps
We don’t need Ops
http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies

14. DevOps is a Re-Org!
- Adrian Cockcroft, Architect, Netflix
RE

15. DevOps Team Patterns
Dev
Ops
Collaboration
Dev
Ops
Embedded
Dev
Ops
DevOps-as-a-service
DevOps
Dev
Ops
DevOps
Temporary DevOps team
http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies

16. Business Case Requirements Use Case Features Plan Go to market
Business
Design Code Refactor
Unit
Test
Bug Fix Deliver
Developers
(application)
Provision Configure Orchestrate Deploy Report Monitor
IT Operations
(infrastructure)
Agile
Development
DevOps
• Continuous Integration
• Continuous Deployment
• IT Automation
• Application Management
• Iterative development
• Scrum, sprints, stories
• Velocity
Business
Agility
IT
Agility
Business and IT agility

17. Collaboration and Communication
Standups
Pairing
Rotation
Visibility

19. Everyone knows what production looks like.

20. Building a DevOps Culture
• Encourage a fail fast, learn quick mindset
• Foster innovation and accountability
• Promote open and honest sharing of lessons
learned
• Build trust across organizational boundaries
- Include all the stakeholders
• Expect that it will be a work-in-progress

21. The DevOps Stack

22. The DevOps Stack
Continuous Deployment
Delivery Pipelines
Deployment Automation
Continuous
Integration
Automated
Testing
Configuration
Management
Agile
Communication

23. • Infrastructure as code
• IT Automation
• Continuous Integration
- Application
§ Compile, test, optimize (code coverage)
- Infrastructure
§ Logical, valid, secure
DevOps Practices

24. • Continuous Deployment
- Application
- Rollout & Rollback
• Version control integration
• Application and Infrastructure version management
• Monitoring and logging
DevOps Practices

25. Continuous Integration
• Every developer check-in initiates a build
- Builds are fully automated
- Tests validate every check-in
- Always check-in to the trunk
• Instant feedback for developers
- The build must remain fast
• Minimizes the impact of broken builds
- Bugs don’t accumulate
• Drives the culture of small, frequent releases

26. Continuous Integration Tools

27. Automated Testing
• Robust, automated testing drives confidence in the
entire process
• Testing is not viewed as a “phase” of a project
- Testing starts on day one
• View failures as successes
• Don’t overlook testing the infrastructure
- Are you ready for the Simian Army?
• Testing is everyone’s responsibility

28. Configuration Management
• Everything must go into version control
• Implement an effective strategy for branching and
merging
• Don’t let tools get in the way

29. Deployment Automation
• Invest in streamlining and unifying all aspects of the
deployment process
• An application’s environment and configuration should be
managed through code
- Remove any dependencies on human intervention
• Every manual step represents an opportunity for
future failure
- Console tweaks = future bugs
• Value repeatability
- How fast can you completely rebuild your environment from scratch?

30. One Step Environment Creation
• Require a single build process for all environments
- Try to build on horizontal concepts
• Expect the environments to be constantly evolving
- Fragile automation becomes fragile environments
• “The longer you wait to have a common environment
build process, the harder it is to create one”

31. Version Control
Build/
Compile
Code
Dev
Unit Test
App Code
IT Ops
DR Env
Test Env
Prod Env
Dev Env
Application
Write
App
Code
Infrastructure
CloudFormation
tar, war, zip
yum, rpm
Deploy
App
Package
Applicatio
n
Deploy application
only
Deploy infrastructure
only
AMI
Build
AMIs
Validate
Template
s
Write
Infra
Code
Deploy
Infras
Automate
Deployme
nt
Artifact Repository
Continuous Integration / Deployment &
Automation

32. Delivery Pipelines
Commit
Unit
Test
System
Test
• Defines the automated lifecycle of delivery
• Enables collaboration between the groups
• Provides everyone with visibility into the flow of change
Commit
Unit
Test
System
Test
UAT Staging ProdCommit
Unit
Test
System
Test

33. Continuous Deployment
• Code that makes it through the pipeline goes straight
to production
• Requires continuous integration and delivery
• Relies heavily on small, constantly tested releases
• Often a stretch goal for many organizations

34. Continuous Deployment vs Delivery

35. • Deploy in-place
- Deploy all at once (Service outage)
- Rolling updates
Deployment Approaches
Demo

36. • Blue-Green Deployment
- Discrete environment
- Multiple environments from branches
- Support A/B testing
- “Rolling DNS”
• Alternate Blue-Green (Red-Black?) deployment
- Attach and detach Auto Scaling groups from Elastic Load balancers
- Avoid messing with DNS
Deployment Approaches

37. Blue-Green Deployment
Amazon
Route 53
EC2 Instances
ELB
100%
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node
• Stand up duplicate
infrastructure and slowly cut
traffic over to it
- Shift via DNS
- Simplifies feature testing
- Simplifies roll back
• As we shift more traffic over,
let auto-scaling grow/shrink
our instances
- Shut down the old when no traffic there

38. Blue-Green Deployment
Amazon
Route 53
EC2 Instances
ELB
EC2 Instances
ELB
90%
10%
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node
• Stand up duplicate
infrastructure and slowly cut
traffic over to it
- Shift via DNS
- Simplifies feature testing
- Simplifies roll back
• As we shift more traffic over,
let auto-scaling grow/shrink
our instances
- Shut down the old when no traffic there

39. Blue-Green Deployment
Amazon
Route 53
EC2 Instances
ELB
EC2 Instances
ELB
0%
100%
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node
• Stand up duplicate
infrastructure and slowly cut
traffic over to it
- Shift via DNS
- Simplifies feature testing
- Simplifies roll back
• As we shift more traffic over,
let auto-scaling grow/shrink
our instances
- Shut down the old when no traffic there

40. Blue-Green Deployment
• Stand up duplicate
infrastructure and slowly cut
traffic over to it
- Shift via DNS
- Simplifies feature testing
- Simplifies roll back
• As we shift more traffic over,
let auto-scaling grow/shrink
our instances
- Shut down the old when no traffic there
Amazon
Route 53
EC2 Instances
ELB
EC2 Instances
ELB
0%
100%
DynamoDB
MySQL RDS
Instance
ElastiCache
Cache Node

41. ASG V1
ELB
Red-Black Deployment
RDS

42. Red-Black Deployment
ELB
ASG V1 ASG V2
UAT
RDS

43. Red-Black Deployment
ELB
ASG V1 ASG V2
RDS

44. Red-Black Deployment
ELB
RDS
ASG V2

45. Agile Everywhere
• Agile tenets are core to all dimensions of the DevOps
lifecycle
- Small, iterative releases
- Quality is everyone’s job
- Customer focused
- Fast, constant feedback
- Done means “released”

46. Automation Everything
• Automation is vital to smaller, more frequent releases
• All steps that add human intervention also add
fragility
- “Small compromises can undermine your success”
• Use knowledge gathered from frequent, automated
releases to constantly evolve the process

47. If you have to login to production, your DevOps-fu is
sub optimal

48. • Scalability (anything manual is not scalable)
• Reliability
• Reproduction/Duplication
• Environment consistency
• Auditability/Record Keeping
• Security
• Governance
Infrastructure as code – why?

49. "WebServer": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"packages" : {
"yum" : {
"httpd" : [],
"php" : [],
"php-mysql" : [],
"php-gd" : [],
"php-xml" : [],
"php-mbstring" : [],
"mysql" : []
}
},
"sources" : {
"/var/www/html" : "http://ftp.drupal.org/files/projects/drupal-7.8.tar.gz",
"/home/ec2-user" : "http://ftp.drupal.org/files/projects/drush-7.x-4.5.tar.gz"
},
AWS
CloudFormation
template
Here’s some infrastructure as Code

50. Describing Infrastructure with Code
Developers
&
Operations
Internal Git
CI Server
Pre-commit
Hook
Testing Environment Subnet
CI Workers
Dev Environment VPC Subnet
DEV WEB
ELBDev Stack
Tier 1
Dev Stack
Tier 2
Dev MySQL
DB Instance
DEV APP
ELB
VPN
TUNNEL
VPN facing VPC Subnet
Internet
Gateway
VPN
Endpoint
Dev Admin
Instance
NAT Instance
Amazon S3
Amazon
DynamoDB
Amazon SQS
Amazon
CloudFront
Amazon
Route 53
This entire infrastructure stack can be constructed, configured, and deployed with code:

51. Infrastructure Code
The code that describes infrastructure should inherit
the same values applied to application code
• Not JUST revision control
• Make use of bug tracking/ticketing systems
• Peer reviews of changes before they happen
• Establish infrastructure code patterns/designs
• Test infrastructure changes like code changes

52. Declarative Approach to:
• Provisioning
• Configuration
• Orchestration
• Reporting
Elastic
Beanstalk
CloudFormation
OpsWorks
Automation and configuration management

53. Automation Tools
AWS Tools
• CloudFormation
• Elastic Beanstalk
• OpsWorks
Third-Party Tools
• Chef
• Puppet

54. Monitoring & Metrics : The DevOps Pulse
• Track every resource possible
• Alert on services, availability, and response times
• Capture, learn, and improve
• Share access with everyone on the team
• Plan metrics and monitoring into the DevOps lifecycle

55. If it moves graph it, or it didn’t happen

56. Monitoring/Logging Tools
• Logging
- Logstash
Ø Check out ELK!
- Sumo Logic
- Splunk
- CloudWatch logs
• Metrics
- CloudWatch
- Ganglia
- StatsD + Graphite + Grafana
- NewRelic
- Monitoring
- Nagios
- Munin
- Sensu
- Hyperion

57. Discussion

58. Thank You