Last update to the DevOps anti-patterns talk that IMO deserves separate upload. It was about anti patterns captured consulting several projects on their DevOps adoption. There are few common pitfalls we can see repeating again and again over DevOps culture discovery. This talk is my experience summary there
3. TALK STRUCTURE
PRESO PLAN
▸ Smell Symptoms: organizational anti patterns for devops
▸ Vitamins: proposed solutions from devops library
▸ Painkillers: deployment anti patterns
▸ Antibiotics: environment anti patterns brought by devops
pic from wonderful “matrix” movie
7. DEFINITION OF DEVOPS
What is DevOps?
▸ Development teams ▸ Operations
▸ Security and compliance ▸ Management
▸ Strategy, IT Strategy
BFG900 from classical game DOOM (c) id software
▸ QA▸ HR
8. DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
9. DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes first
▸ cross functional
▸ software defined
data centers
▸ complex communication ▸ complexity theory
11. ANTIPATTERN # 0
BI MODAL IT
Sparky from Frankenweenie
▸ Slow IT
▸ Safe and Predictable
▸ IT Centric
▸ Secure and Regulated
▸ Fast IT
▸ Fast and Agile
▸ Business Centric
▸ Innovative
12. ANTIPATTERN # 1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certified DevOps master
▸ Give me DevOps compliance checklist
▸ Give me 5 key DevOps control metrics
13. ANTIPATTERN # 1.1
REBRANDED IT
DevOps != Configuration Management
DevOps != Release Management|
DevOps != Product Management
…
DevOps != (*) Management
DEVOPS IS THE CULTURE YOU
CANNOT FIND IN IN ITIL CATALOG
14. ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
You don’t need to change
anything in your org
You extend organization
with new capabilities
15. ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
Creates Unicorn vs Horses dilemma
DevOps builds their own Silo
Often loses focus on value
and speed
16. ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸ RTFM Architecture
17. ANTIPATTERN # 2: PROPOSED SOLUTION
SORRY, NOT MY DEPARTMENT
Bread ownership and
specialization with
autonomous teams
vs
http://martinfowler.com/bliki/DevOpsCulture.html
18. ANTIPATTERN # 3
DEVOPS IN AGILE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”
20. ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance
manually leads to unique and distinct
server configuration footprint
(TECHNICAL DEBT)
http://martinfowler.com/bliki/SnowflakeServer.html
28. VITAMINS
USE CI ENVIRONMENT
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸ Don’t judge for broken builds
▸ Go home when build is green
33. ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
With single master CI you easily get a
single point of failure
CI master
34. ANTIPATTERN # 6: PROPOSED SOLUTION
MULTI MASTER “JENKINS”
Multi-master CI can dedicate CI Master
per group of related components
CI master CI master CI master
35. ANTIPATTERN # 6: PROPOSED SOLUTION # 2
SHARE RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Implement Pipeline as Code
▸ Store CI/CD pipelines in git
▸ Let pipeline evolve together with your app
37. ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
38. ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
Time adjustable
if TREND is good
39. ANTIPATTERN # 7: PROPOSED SOLUTION
AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST B
feedback
feedback
measure
51. ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning
Chnorr Service
52. ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS Chnorr Service
53. ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
Chnorr Service
54. ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
Chnorr Service
55. ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
cmdb
Chnorr Service
56. ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
57. ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
58. ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
59. ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure
code
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
60. ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
Chnorr Service
61. ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
Chnorr Service
62. ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
complicated?
Chnorr Service
63. ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
ConfigureHarden Download Install
Chnorr Service
64. ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
Chnorr Service
65. ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
code
packer
PROVISION
MAKE
SNAPSHOT
docker
Chnorr Service
RECONFIGURE
66. ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure
code
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
Chnorr Service
67. ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure
code
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
▸ Version of your infra code
▸ Maintain dependencies
Chnorr Service
68. ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept Tests
CI
Unit Test Provision
…
PRECISE version libs
Accept Tests
CI
Unit Test Provision
…
LATEST version libs
Chnorr Service
Chnorr Service
69. ANTIPATTERN # C
INFRASTRUCTURE PET
Attributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual nursing
▸ Requires scary patching
▸ Leads to snowflakes
▸ PaaS is modern pet
70. ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLE
Attributes of Pet
▸ Have numbers in its name
▸ Short living instance
▸ Immutable configuration
▸ Recreate instead of patching
▸ Requires careful planning
71. ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
72. ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
73. ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
74. ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
User Data
75. ANTIPATTERN # D
SECRETS LEAK
OS
Configure
Infrastructure
code
Install
cmdb
wrong place
for your secrets
wrong place
for your secrets
Chnorr Service
76. ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with configuration
▸ Don’t leave secrets in service
Secrets DON’Ts
77. ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your fiend
Secrets DOs
78. ANTIPATTERN # D: PROPOSED SOLUTION
SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
AWS
87. TAKEAWAYS
READING
▸ Book: A Human Error Approach to
Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann
Scott A. Shappell
▸ ISBN: 978-0754618737
88. THANK YOU
Download me here: http://www.slideshare.net/akranga/dev-ops-with-smell-v12
Twitter: @acankr