At Zalando we have 265 AWS accounts and 31 Kubernetes clusters running in AWS. We have to make sure that feature teams do not have to do more work than needed to deploy their application and publish endpoints. Creating a Kubernetes ingress object is enough to provision an AWS Application Load Balancer (ALB) including automated SSL certificate lookup and TLS termination, HTTP routing based on our ingress implementation and a public resolvable DNS entry. This talk will explain the stack and deep dive into how this works for us.

1. DOCKER MEETUP BERLIN
2017-07-19
SANDOR SZÜCS
@sszuecs
Automatic
infrastructure for
Kubernetes ingress
in AWS

2. 2
ZALANDO
15 markets
6 fulfillment centers
20 million active customers
3.6 billion € net sales 2016
165 million visits per month
12,000 employees in Europe

3. 3
ZALANDO TECHNOLOGY
HOME-BREWED,
CUTTING-EDGE
& SCALABLE
technology solutions
>1,700
employees from
tech locations
+ HQs in Berlin6
77
nations
help our brand to
WIN ONLINE

4. 4
ZALANDO TECH’S
INFRASTRUCTURE

5. 5
FOUR ERAS AT ZALANDO TECH
ZOMCATPHP STUPS KUBERNETES
2010 2015 2016
Data center
WAR
LXC
AWS
Docker
Cloud Formation
Low level (AWS API)
AWS
Docker
Cloud Formation
Kubernetes manifest
Higher abstraction level
Data center
PHP files

6. 6
LARGE SCALE?

9. 9
KUBERNETES:
ARCHITECTURE

10. 10
ISOLATED AWS ACCOUNTS
Internet
*.abc.example.org *.xyz.example.org
Product ABC Product XYZ
EC2
LBLB

11. 11
KUBERNETES ON AWS

12. 12
DEPLOYMENT

13. 13
DEPLOYMENT CONFIGURATION
.
├── apply
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml # K8s Ingress
│ ├── kube-secret.yaml # K8s Secret
│ └── kube-service.yaml # K8s Service
├── deployment.yaml # K8s Deployment
└── pipeline.yaml # CI/CD config

14. 14
INGRESS.YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "myapp.foo.example.org"
http:
paths:
- backend:
serviceName: "myapp"
servicePort: 80

15. 15
JENKINS DEPLOY PIPELINE

16. 16
CHALLENGES

17. 17
1. Getting Started
2. Ingress
CHALLENGES

18. 18
CHALLENGE 1:
GETTING STARTED

19. 19
GETTING STARTED
https://github.com/hjacobs/kubernetes-on-aws-users

20. 20
GETTING STARTED
https://github.com/hjacobs/kubernetes-on-aws-users

21. 21
CLUSTER PROVISIONING
• Two Cloud Formation stacks
• Master & worker ASGs + etcd
• Nodes w/ Container Linux
• K8s manifests applied separately
• kube-system Deployments
• DaemonSets

22. 22
CLUSTER PROVISIONING

23. 23
GETTING STARTED
Other questions we asked ourselves..
• Single AZ vs. Multi AZ? ⇒ Multi AZ
• Federation? ⇒ No, not ready yet
• Overlay network? ⇒ Flannel, “rock solid”
• Authnz? ⇒ OAuth, webhook

24. 24
CHALLENGE 2:
Ingress

25. 25
Ingress
• System view
• Developer point of view
• Enhancing Kubernetes with
weighted traffic switching

26. 26
System View
Goal: use Kubernetes API as primary interface
• External DNS → Route53
• Kubernetes Ingress Controller for AWS → ALB+TLS
• Skipper - http router → your dockerized app
⇒ we wrote new components
to achieve our goal

27. 27
System View - traffic flow
https://github.com/zalando/skipper
ALB
Node Skipper Node Skipper
MyApp MyApp MyApp
Service Service
K8s network
EC2 network
TLS
HTTP

28. 28
Kubernetes - ingress - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns

29. 29
Kubernetes - service - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
Select PODs by label
Target of Ingress definition
Port of the my-app

30. 30
Kubernetes - POD - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
Target port
Selected by service

31. 31
Skipper
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
• Skipper used as main shop router
• Skipper can be used as ingress implementation

32. 32
Skipper
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns

33. 33
System View - skipper - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
Skipper reads

34. 34
kube-ingress-aws-controller
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns

35. 35
System View - controller - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
kube-ingress-aws-controller
Optional
read
write

36. 36
System View - external DNS
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns

37. 37
System View - external-dns - configuration
https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
external-dns
reads

38. 38
System View - summary
● DNS Name pointing to ALB
● ALB + TLS
● Skipper routes

39. 39
Developer point of view
● Define templates
● Execute pipelines

40. 40
Developer point of view - defines template
Pipeline expands some
variables from
pipeline.yaml

41. 41
Developer point of view - executes pipeline

42. 42
WEIGHTED TRAFFIC SWITCHING
• STUPS - former AWS platform toolkit
• Kubernetes
• supports rolling updates
• enhancement to ingress required

43. 43
WEIGHTED TRAFFIC SWITCHING - STUPS
• STUPS uses weighted Route53 DNS records
• Allows canary, blue/green, slow ramp up
• Problem: DNS caching

44. 44
WEIGHTED TRAFFIC SWITCHING - Kubernetes
• Approach: add weights to Ingress backends
https://github.com/zalando/skipper

45. 45
Traffic Switching - Ingress configuration
https://github.com/zalando/skipper
Skipper reads

46. 46
LINKS
Running Kubernetes in Production on AWS
http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html
Kube AWS Ingress Controller
https://github.com/zalando-incubator/kube-ingress-aws-controller
Skipper HTTP Ingress Router
https://github.com/zalando/skipper/
External DNS
https://github.com/kubernetes-incubator/external-dns
Zalando Cluster Configuration
https://github.com/zalando-incubator/kubernetes-on-aws
PostgreSQL Operator
https://github.com/zalando-incubator/postgres-operator

47. QUESTIONS?
SANDOR SZÜCS
TECH INFRASTRUCTURE
CLOUD ENGINEER
sandor.szuecs@zalando.de
@sszuecs
Slides based on @try_except_
Illustrations by @01k