While the adoption of DevOps makes teams move faster with reduced dependency on central operations, it can constrain teams who lack the skills to self-manage the full application and infrastructure stack. The way to overcome this challenge is creating an internal platform and treating it as a world-class product offering. “Applying product management to internal platforms means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience”, via ThoughtWorks Technology Radar. In this talk, Henning Jacobs will walk you through how Zalando adopted a customer-first mindset with regards to its developer tooling. He will show the effect on developer satisfaction when internal platforms are given the same respect as external product offerings. Henning will furthermore tell his story about how Zalando moved from a classical infrastructure team to a product mindset with strong focus on building a world-class developer experience. Henning shares both their learnings and challenges going through this transition, and the impact it has on the daily life of Zalando’s customers (developers). This talk was given in Aarhus on 4th of June 2019.

1. Why we don't use
the term DevOps
DESTINATION AARHUS
2019-06-04
HENNING JACOBS
@try_except_

2. 2
EUROPE’S LEADING ONLINE FASHION PLATFORM

3. 3
ZALANDO AT A GLANCE
~ 5.4billion EUR
revenue 2018
> 250
million
visits
per
month
> 15.000
employees in
Europe
> 79%
of visits via
mobile devices
> 26
million
active customers
> 300.000
product choices
~ 2.000
brands
17
countries

4. 4
THE BIRTH OF DEVOPS - 2009
“10+ Deploys per Day: Dev and Ops Cooperation at Flickr.”
DevOps Days Belgium
https://devops.com/the-origins-of-devops-whats-in-a-name/

5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

7. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

8. 8
THE PHOENIX PROJECT - 2013
"The Three Ways"
1. Systems Thinking
2. Amplify Feedback Loops
3. Culture of Continual
Experimentation And Learning

9. 9
WHAT HAPPENED NEXT

10. 10
CARGO CULTING

11. 11
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

12. 12
DevOps
is normal

13. 13

14. 14
WHAT HAPPENED NEXT

15. 15
WHAT HAPPENED NEXT

16. 16
DEV + OPS ⇒ DEVOPS
Devs
"DevOps"
Team
Ops

17. 17
DEV + OPS ⇒ ENGINEERING
Devs
"DevOps"
Team
Ops
Engineering
Team

18. 18
PRODUCT DEVELOPMENT TEAMS
"Great products emerge from
empowered, fully cross-functional teams."
- John Cutler, 2019
https://amplitude.com/blog/on-being-product-led

19. 19
DEVOPS IS
It is a culture, in which people
work together to improve the
product delivery cycle.

20. 20
DEVOPS ANTI-PATTERNS
Dev and Ops Silos
DevOps Team Silo
Dev Don't Need Ops
DevOps as Tools Team
Rebranded SysAdmin
Ops Embedded in Dev Team
web.devopstopologies.com

21. 21

22. 22
TEAM INTERACTION MODES
• Collaboration: 2 teams working together
• X-as-a-Service: 1 provides, 1 consumes
• Facilitating: 1 team helps another
Product teams need a family too - DevOpsDays Portugal

23. 23
A BRIEF HISTORY OF
ZALANDO TECH

24. 24
2010
"Sysop-Test"
"QA-Test"

25. 25
DEVOPS IS ABOUT SKIN IN THE GAME
Absence of skin in the game:
People who are isolated from the
impacts of their decisions do not learn.
They remain captive to their
erroneous ideas about
how the world works.

26. 26
2013: SELF SERVICE

27. 27
2015: RADICAL AGILITY
AWS
STUPS
DOCKER
DEPLOY
SSH
ACCESS
AUDIT
REPORTS
FULL AWS
ACCESS
Teams have
admin access
& full
responsibility

28. 28
2015: ISOLATED AWS ACCOUNTS
Internet
*.abc.example.org *.xyz.example.org
Team ABC Team XYZ
EC2EC2
ELBELB
EC2

29. 29
Platform
> 1100
developers
> 200
development teams
2019

30. 30
YOU BUILD IT, YOU RUN IT
The traditional model is that you take your software to the
wall that separates development and operations, and
throw it over and then forget about it. Not at Amazon.
You build it, you run it. This brings developers into
contact with the day-to-day operation of their software. It
also brings them into day-to-day contact with the
customer.
- A Conversation with Werner Vogels, ACM Queue, 2006

31. 31
ON-CALL: YOU OWN IT, YOU RUN IT
When things are broken,
we want people with the best
context trying to fix things.
- Blake Scrivener, Netflix SRE Manager

32. 32
"SKIN IN THE GAME"

33. 33
DEVELOPER JOURNEY
Consistent story
that models
all aspects of SW dev

34. 34
Developer
Journey

35. 35
Developer
Journey
Correctness
Compliance
GDPR
Security
Cost Efficiency
24x7 On Call
Governance
Resilience
Capacity
...

36. 36
DEVELOPER PRODUCTIVITY
Code Build Test Deploy OperateSetup
Cloud Native Application Runtime

37. 37
CLOUD NATIVE
.. uses an open source software stack to deploy
applications as microservices, packaging each part into
its own container, and dynamically orchestrating those
containers to optimize resource utilization.
Cloud native technologies enable software developers to
build great products faster.
- https://www.cncf.io/

38. 38
CONTAINERS END-TO-END
Code Build Test Deploy OperateSetup
Cloud Native Application Runtime

39. 39
CONTAINERS

40. 40
CONTAINERS

41. 41
PLATFORM AS A PRODUCT
"A good platform is treated as a product
(reliable, usable, fit for purpose)."
- Manuel Pais, 2019
Product teams need a family too - DevOpsDays Portugal

43. 43
PLAN & SETUP

44. 44
Plan
Stories
Rules of Play
Tech Radar

46. 46
Setup
Application
Bootstrapping

49. 49
BUILD & TEST

50. 50
CDPGit
code
push
CONTINUOUS DELIVERY PLATFORM: BUILD

52. 52
DEPLOY

53. 53
Deploy
Kubernetes

54. 54
DEPLOYMENT CONFIGURATION
├── deploy/apply
│ ├── deployment.yaml
│ ├── credentials.yaml # Zalando IAM
│ ├── ingress.yaml
│ └── service.yaml
└── delivery.yaml # Zalando CI/CD

55. 55
INGRESS.YAML
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "myapp.foo.example.org"
http:
paths:
- backend:
serviceName: "myapp"
servicePort: 80

56. 56
TEMPLATING: MUSTACHE
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "{{{APPLICATION}}}.example.org"
http:
paths:
- backend:
serviceName: "{{{APPLICATION}}}"
servicePort: 80

57. 57
CONTINUOUS DELIVERY PLATFORM

58. 58
CDP: DEPLOY
"glorified kubectl apply"

59. 59
CDP: OPTIONAL APPROVAL

60. 60
STACKSET: TRAFFIC SWITCHING
github.com/zalando-incubator/stackset-controller

61. 61
STACKSET CRD
kind: StackSet
...
spec:
ingress:
hosts: ["foo.example.org"]
backendPort: 8080
stackLifecycle:
scaledownTTLSeconds: 1800
limit: 5
stackTemplate:
spec:
podTemplate:
...
github.com/zalando-incubator/stackset-controller

62. 62
TRAFFIC SWITCHING STEPS IN CDP
github.com/zalando-incubator/stackset-controller

63. 63
EMERGENCY ACCESS SERVICE
Emergency access by referencing Incident
zkubectl cluster-access request
--emergency -i INC REASON
Privileged production access via 4-eyes
zkubectl cluster-access request REASON
zkubectl cluster-access approve USERNAME

64. 64
INTEGRATIONS

65. 65
CLOUD FORMATION VIA CI/CD
├── deploy/apply
│ ├── deployment.yaml # Kubernetes
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml
│ ├── kube-secret.yaml
│ └── kube-service.yaml
└── delivery.yaml # CI/CD config
"Infrastructure as Code"

66. 66
ZALANDO IAM/OAUTH VIA CRD
kind: PlatformCredentialsSet
..
spec:
application: my-app
tokens:
read-only:
privileges:
- com.zalando::foobar.read
clients:
employee:
grant: authorization-code
realm: users
redirectUri: https://example.org/auth/callback
Extensible Kubernetes API

67. 67
POSTGRES OPERATOR
Application to manage
PostgreSQL clusters on
Kubernetes
>700
clusters running
on Kubernetes
github.com/zalando/postgres-operator

68. Elasticsearch in Kubernetes
Elasticsearch
2.500 vCPUs
1 TB RAM
github.com/zalando-incubator/es-operator/

69. 69
SUMMARY
• Application Bootstrapping
• Git as source of truth and UI
• 4-eyes principle for master/production
• Extensible Kubernetes API as primary interface
• OAuth/IAM credentials
• PostgreSQL
• CloudFormation for proprietary AWS services

70. 70
DELIVERY PERFORMANCE METRICS
• Lead Time
• Release Frequency
• Time to Restore Service
• Change Fail Rate
https://srcco.de/posts/accelerate-software-delivery-performance.html

71. 71
CONTAINERS
From "Accelerate: The Science of Lean Software and DevOps"

72. 72
DELIVERY PERFORMANCE METRICS
• Lead Time
• Release Frequency
• Time to Restore Service
• Change Fail Rate
≙ Commit to Prod
≙ Deploys/week/dev
≙ MTRS from incidents
≙ n/a

73. “.. means establishing empathy with internal
consumers (read: developers) and collaborating
with them on the design. Platform product managers
establish roadmaps and ensure the platform delivers
value to the business and enhances the developer
experience.”
- ThoughtWorks Technology Radar

74. 74
PRODUCT MINDSET
... means we are developing all our products
for the best customer experience
and biggest business impact
while all actions are
aligned with our strategy.

76. 76
DEVELOPER SATISFACTION

77. 77
DOCUMENTATION
"Documentation is hard to find"
"Documentation is not comprehensive enough"
"Remove unnecessary complexity and obstacles."
"Get the documentation up to date and prepare
use cases"
"More and more clear documentation"
"More detailed docs, example repos with more
complicated deployments."

78. 78
DOCUMENTATION
• Restructure following
www.divio.com/en/blog/documentation/
• Concepts
• How Tos
• Tutorials
• Reference
• Global Search
• Weekly Health Check: Support → Documentation

80. 80
NEWSLETTER
"You can now.."
• You can now benefit from the most recent
Kubernetes 1.12 features, e.g. ..
• You can now analyse your Kotlin project with
SonarQube and upload your Scala code coverage
report to SonarQube

81. 81
SIGNAL: ISSUE UPVOTES

82. 82
TESTIMONIALS
“Useful information, good level of details and pleasant to
read. It's one of the few newsletters that I took time to
read entirely :)”
- a reader, July 2018

83. 83
TESTIMONIALS
“So, thank you, Team Automata, for listening to our
community, taking our upvotes in consideration when
developing new solutions and building every day
'the first CI that doesn't suck'.”
- a user, October 2018

84. 84
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

85. 85
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

86. 86
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

87. 87
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

88. 88
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

89. 89
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

90. 90
DevOps
is normal

91. 91
DevOps
is normal

92. 92
You build it,
you run it
is normal

93. 93
WAIT A SECOND..
What about
my job title?

94. 94
"DEVOPS ENGINEER" AS A JOB TITLE

95. 95
T-SHAPED ENGINEER
"Generalizing specialist"
Breadth of Knowledge
TechnicalDepth

96. 96
DevOps Engineer
Software Engineer /
Software Developer
(T-Shaped)

97. 97

98. 98
OPEN SOURCE & MORE
Kubernetes on AWS
github.com/zalando-incubator/kubernetes-on-aws
Skipper HTTP Router & Ingress controller
github.com/zalando/skipper
External DNS
github.com/kubernetes-incubator/external-dns
Postgres Operator
github.com/zalando-incubator/postgres-operator
More Zalando Tech Talks
github.com/zalando/public-presentations

99. QUESTIONS?
HENNING JACOBS
HEAD OF
DEVELOPER PRODUCTIVITY
henning@zalando.de
@try_except_
Illustrations by @01k