What we've built at Zalando is complex. Supporting – profitably – a publicly traded ecommerce company that does business in 15 diverse European markets, with more than 16 million active users who all speak different languages, use different payment methods, prefer different shipping methods, and have different product tastes, has required nonstop innovation. Until recently we've focused on building a unified, comprehensive retail system, quickly, that solves just our problems. But to truly fight against complexity--particularly the accidental complexity that slows down our development process--we have adopted a microservices architecture. And when it comes to DevOps, we’ve gone a step beyond the "You build it, you run it" motto--working in autonomous teams with DevOps treated as a "first-class entity.” In this talk, Jan Löffler (Head of Platform Engineering) will share Zalando's experience implementing “Radical Agility” from a DevOps perspective. “Radical Agility” is the Zalando technology team’s multi-pronged approach to managing the complexity that results from building an architecture of massive size. Jan will focus on how microservices enable Zalando’s engineers to move faster and build systems that scale, at scale, and avoid dependencies. He will show how microservices, in conjunction with a cloud infrastructure, support teams as they try strive for autonomy. Finally, he will draw upon his experiences to show how this all works in practice, and discuss what is organizationally and architecturally necessary to make DevOps a top priority for all members of your tech organization.

1. Radical Agility
with Autonomous Teams and Microservices
jan.loeffler@zalando.de / @jlsoft2
GOTO Copenhagen 2015-10-06

2. We shape our buildings;
thereafter they shape us

3. Conway’s Law
“organizations which design
systems ... are constrained
to produce designs which
are copies of the
communication structures of
these organizations”
Melvin Conway

4. AN
ARCHITECTURE
FOR
INNOVATION

9. A BRIEF
HISTORY OF
ZALANDO
TECHNOLOGY

13. October
900+
Apps
800+
Tech employees

14. Platform Platform team
request servers
deploy

15. Platform
70+ delivery teams
Platform team
deploy
request servers
request storage

18. DELIVER AMAZING
PRODUCTS
EFFICIENTLY AT
SCALE, AND
FEELING GREAT
ABOUT IT.

19. PURPOSE - AUTONOMY - MASTERY

23. DRIVE
The Surprising Truth
About What
Motivates Us
Daniel Pink

24. FROM
CONTROL &
COMMAND
TO PURPOSE
AND TRUST

25. DELIVERY
LEAD
PEOPLE
LEAD

26. GLOBAL
REGRESSION
INNOVATION LAB
TECH
SERVICE
PRODUCT
OWNER
PRODUCT
SPECIALIST
DELIVERY
LEAD
PEOPLE
LEAD
BUSINESS
ASSURANCE
PRODUCT
OVERARCHING
ADMIN & SUPPORT
CONTROLLING
EXECUTIVE
SUPPORT
COMPLIANCE
RISK ,
SECURITY &
STRATEGY
ONBOARDING &
TECHADEMY
AGILE
COACHING PROJECT
MANGEMENT
ENGINEERING
PRODUCTIVITY
DELIVERY
LEAD
PEOPLE LEAD
DELIVERY

28. OKR

30. API FIRST

31. REST

32. SAAS

33. MICRO
SERVICES

34. CLOUD

35. OPEN
SOURCE

37. Compliance Innovation

38. file:///Users/kwalckermaye/Downloads/Mobil
e-Developers-look-ov-008.jpg
file:///Users/kwalckermaye/Downloads/deskt
op_death-600x369.jpg
file:///Users/kwalckermaye/Downloads/07235
8-wired.gif.jpeg
file:///Users/kwalckermaye/Downloads/the-
death-of-the-desktop.jpg

40. WHERE
TO GO
DataCenter I DataCenter II AWS
APP
1
APP
2
APP
3
APP
4
APP
5
APP
6
APP
1
APP
2
APP
3
APP
4
APP
5
APP
6
APP
1
APP
2
APP
3
APP
4

42. STUPS.io
STUPS To Unleash Penguin Swarms

44. One AWS account per Team
Deployment with Docker
Managed SSH Access
REST / OAuth 2.0 mandatory

45. AWS
STUPS
DOCKER
DEPLOY
SSH
ACCESS
AUDIT
REPORTS
FULL AWS
ACCESS

46. Internet
*.abc.example.org *.xyz.example.org
Team ABC Team XYZ
EC2EC2
ELBELB
EC2

47. ELB myapp-1
myapp.example.org
EC2
+ Docker
EC2
+ Docker
EC2
+ Docker

48. ELB myapp-1
EC2
+ Docker
EC2
+ Docker
EC2
+ Docker
ELB myapp-2
EC2
+ Docker
EC2
+ Docker
myapp.example.org

49. ELB myapp-2
EC2
+ Docker
EC2
+ Docker
myapp.example.org

50. SCM
Ticket system
Issue “ABC-123”
Commit “afb123”
msg: ABC-123..

51. SCM
Pier One Docker Registry
build
Ticket system
Image “docker/myart:1.0”
commit: afb123Issue “ABC-123”
Commit “afb123”
msg: ABC-123..

52. Pier One Docker Registry
build
approved
Application Version “1.0”
artifact: docker/myart:1.0
Ticket system
Application Registry
SCM
Image “docker/myart:1.0”
commit: afb123Issue “ABC-123”
Commit “afb123”
msg: ABC-123..
✓ Specification
✓ Artefact tested

53. Pier One Docker Registry
build
approved
EC2 Instance
Docker
Container
Application Version “1.0”
artifact: docker/myart:1.0
AMI
Ticket system
Application Registry
SCM
Image “docker/myart:1.0”
commit: afb123Issue “ABC-123”
Commit “afb123”
msg: ABC-123..
✓ Specification
✓ Artefact tested

54. AWS
Senza CLI
Docker
Registry
docker pull
docker push
AMI

55. AWS
Developer
Console
get access
token
AMI
Password
Rotator
OAuth
Provider
store
passwords
get password
S3
rotate
passwords
Application
Registry

57. Spilo
Highly available open-source
PostgreSQL appliance
https://github.com/zalando/spilo

59. Try STUPS.io
https://stups.io
https://docs.stups.io
All components are Open Source
https://github.com/zalando
https://github.com/zalando-stups

60. Jan Löffler
● Head of Platform Engineering
● Twitter: @jlsoft2
● jan.loeffler@zalando.de
● http://www.slideshare.net/jlsoft/

61. We shape our buildings;
thereafter they shape us

62. STUPS website & docs
https://stups.io
https://docs.stups.io
All components are Open Source
https://github.com/zalando
https://github.com/zalando-stups

63. BACKUP

66. FROM zalando/openjdk:8u40-b09-4
EXPOSE 8080
COPY target/hello-world.jar /
COPY target/scm-source.json /
CMD java $(java-dynamic-memory-opts) ↲
-jar /hello-world.jar
DOCKERFILE

67. $ docker build -t ↲
pierone.example.org/myteam/hello-world:0.2 .
$ pierone login
Getting OAuth2 token "pierone".. OK
Storing Docker client configuration in ~/.dockercfg.. OK
$ docker push pierone.example.org/myteam/hello-world:0.2
DOCKER BUILD & PUSH

68. $ pierone tags myteam hello-world
Team │Artifact │Tag │Created│By |
myteam hello-world 0.1-andre-test 13d ago ahartmann
myteam hello-world 0.1 3d ago ahartmann
myteam hello-world 0.2 3m ago hjacobs
$ pierone scm myteam hello-world 0.2
Tag│Author │URL │Revision │Status│Created│By |
0.2 hjacobs git:git@github.. 442b7502 10m ago hjacobs
VERIFY IMAGE UPLOAD

69. SENZA: DEFINITION YAML
SenzaInfo:
StackName: hello-world
Parameters:
- ImageVersion:
Description: "Docker image version of Hello World."
SenzaComponents:
- Configuration:
Type: Senza::StupsAutoConfiguration # auto-detect network setup
- AppServer: # will create a launch configuration and ASG with scaling triggers
Type: Senza::TaupageAutoScalingGroup
InstanceType: t2.micro
SecurityGroups: [app-hello-world]
ElasticLoadBalancer: AppLoadBalancer
TaupageConfig:
runtime: Docker
source: "stups/hello-world:{{Arguments.ImageVersion}}"
ports:
8080: 8080

70. SENZA: STACK DEPLOYMENT
$ senza create hello-world.yaml 1 0.2
Generating Cloud Formation template.. OK
Creating Cloud Formation stack hello-world-1.. OK
$ senza events hello-world.yaml 1
Stack Name│Ver.│Resource Type │Resource ID │Status │Status Reason │Event Time
hello-world 1 CloudFormation::Stack hello-world-1 CREATE_IN_PROGRESS User Initiated 10m ago
...
hello-world 1 CloudFormation::Stack hello-world-1 CREATE_COMPLETE 6m ago

71. docker run -d --log-driver=syslog ↲
--restart=on-failure:10 ↲
-e DB_SUBNAME=.. ↲
-v /meta:/meta:ro ↲
-e CREDENTIALS_DIR=/meta/credentials ↲
-p 8080:8080 -p 7979:7979 ↲
-u 999 ↲
pierone.example.org/stups/pierone:0.5
TAUPAGE: DOCKER COMMAND LINE

72. SENZA: MANAGE STACKS

74. docker run .. --log-driver=syslog ..
/etc/rsyslog.d/24-application.conf
:syslogtag, startswith, "docker" ↲
/var/log/application.log
/etc/logrotate.d/..
Don’t forget log rotation..
TAUPAGE: DOCKER SYSLOG

75. APPLICATION LOGS: TAUPAGE SUPPORTS LOGENTRIES AND SCALYR

77. SSH ACCESS: TIME-LIMITED ACCESS TO ANY TEAM SERVER

79. TODO: Screenshot
ZMON

80. ZMON APPLIANCE
*.foo.example.org *.bar.example.org
Team “Foo” Team “Bar”
EC2
Instance
EC2
InstanceEC2
Instance
EC2
Instance
ZMON
Appliance
ZMON
Appliance
KairosDB
EC2
Instance
EC2
Instance
ZMON
Controller
ELB ELB

81. HYSTRIX TURBINE

83. Ubuntu & OpenJDK base image
Log to STDOUT
Config via environ. vars (+ KMS decryption)
Non-root execution
Persistence via EBS mounts
Immutable stacks, no orchestration
DNS endpoints, etcd e.g. for Hystrix streams
RECAP: DOCKER IN STUPS

84. Securing REST based Microservices
Lock all doors
● Open only ports that you need
permanently
● Grant SSH access only temporary
● Use a firewall for each app container
Encrypt all channels
● Keep all tranferred data confidential
● Use TLS
Authenticate & Authorize
● Make sure you talk to the right guy
● Check if access is permitted
● Use OAuth 2.0
Monitor & Limit your API calls
● Control who accesses your API how
often, when and from where
● Use Rate Limiting against fraud
● Use external Security Services against
DDoSIsolate your apps
● Only one app per container / server
● Use Docker or CoreOS Rkt

85. OAuth 2.0 explained
User “Jan”
Browser
Resource Server
(App Server)
Authorization Server
(OpenAM / OpenDJ)
I want to see all my past orders!
Hey Order API, could you please list all past orders of Jan Loeffler?
Sorry, this resource is protected! Please show me your Access Token!
Hi, could you please show me your credentials? I need to check your identity.
Sure, I am jan.loeffler@zalando.de and my password is secret.
Hey Order API, here is my access token:
d2fa5d27-2acc-4b06-95i8-6d3018d94b4f
Hey DJ, I was just given this token:
d2fa5d27-2acc-4b06-95i8-6d3018d94b4f
To whom does it belong?
Sure, the token belongs to
jan.loeffler@zalando.de and is still valid.
So, here are all orders of Jan Loeffler.
Here are all all your orders!