Builders' Day Presentation

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

2. Martin Dominguez
Solutions Architect
AWS
Mastering Kubernetes on AWS

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Kubernetes
• AWS
• Mastering

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
$ vi Dockerfile
$ docker build -t mykillerapp:0.0.1 .
$ docker run -it mykillerapp:0.0.1
Running containers in development is easy…

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving to production: data plane
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
Server
Guest OS
AZ 1 AZ 2
AZ 3

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Moving to production: control plane
etcd etcdetcd
Master Master Master
Availability zone 1 Availability zone 2 Availability zone 1

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Yup. This is hard.
- Lucas Käldström, volunteer ambassador for the Cloud Native Computing Foundation

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Run Kubernetes for me.”
51%
of Kubernetes
workloads run on AWS
today
— Cloud Native Computing Foundation

9. “Give us an upstream experience.”
“Please don’t fork.”
“Make sure it’s compatible”

10. Amazon EKS is Kubernetes Certified

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS
mycluster.eks.amazonaws.com
EKS Workers
Kubectl
AZ 1 AZ 2 AZ 3
VPC

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross account Kubernetes
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• All the pods can communicate with each other directly
without NAT
• All the nodes can communicate with all pods (and vice versa)
without NAT
• The IP that a pod sees itself as is the same IP that others see it
as
The three rules of Kubernetes networking…

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS VPC CNI plugin
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet – 10.0.0.0/24
Instance 1 Instance 2
VPC

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS VPC CNI plugin – understanding IP allocation
• primary CIDR range è RFC 1918 addresses è 10/8, 172.16/12, 192.168/16
Used in EKS for:
• Pods
• Cross-account ENIs for (masters à workers) communication (exec, logs, proxy
etc.)
• Internal Kubernetes services network (10.100/16 or 172.20/16 – chosen based on
your VPC range)
Setup:
• EKS cluster creation è provide list of subnets (in at least 2 AZs!) è tagging

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS VPC CNI plugin – understanding IP allocation
• secondary CIDR ranges (new!) è non-RFC 1918 address blocks (100.64.0.0/10
and 198.19.0.0/16)
Used in Amazon EKS for:
• Pods only
How?
• Amazon EKS custom network config è enable è create ENIConfig CRD è
annotate nodes
CNI
1.2.1+

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Exposes the service on a cluster-internal IP
• Only reachable from within the cluster
• Access possible via kube-proxy
• Useful for debugging services, connecting from
your laptop or displaying internal dashboards
Kubernetes ServiceType: ClusterIP

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes ServiceType: NodePort
• Exposes the service on each Node’s IP at a
static port.
• Routes to a ClusterIP service, which is
automatically created.
• from outside the cluster:
<NodeIP>:<NodePort>
• 1 service per port
• Uses ports 30000-32767

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Exposes the service externally using a cloud
provider’s load balancer.
• NodePort and ClusterIP services (to which LB
will route) automatically created.
• Each service exposed with a LoadBalancer (ELB
or NLB) will get its own IP address
• Exposes L4 (TCP) or L7 (HTTP) services
Kubernetes ServiceType: LoadBalancer

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service load balancer: NLB
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service load balancer: NLB
• NLB supports forwarding the client’s IP through to the node
• .spec.externalTrafficPolicy = Local è client ip passed to pod
• Nodes with no matching pods will be removed by specified NLB’s health check
.spec.healthCheckNodePort
• Use DaemonSet or pod anti-affinity to verify even traffic split

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Maps: service =>
CNAME(externalName field)
• No proxying
• Accessing my-service works in the
same way as other Services
• redirection happens at the DNS level
(rather than via proxying or
forwarding)
kind: Service
apiVersion: v1
metadata:
name: my-service
namespace: prod
spec:
type: ExternalName
externalName:
my.database.example.com
Kubernetes ServiceType: ExternalName

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• exposes HTTP/HTTPS routes
to services within the cluster
• Many implementations: ALB,
Nginx, F5, HAProxy etc
• Default Service Type:
ClusterIP
Kubernetes Ingress Object

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ALB Ingress Controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Implementing logging with EFK
fluentd is an open source
data collector providing a
unified logging layer
elasticsearch is a
distributed, RESTful search
and analytics engine
kibana lets you visualize
your Elasticsearch data

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Implementing logging with EFK - DIY
EKS Worker
pod
fluentd
daemonset

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Implementing logging with EFK - DIY
EKS Worker
pod
fluentd
daemonset

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer Story: Snap

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Snap’s Approach to Infrastructure
Goals
Flexibility Security Availability /
Performance
Cost Reduction Minimize
operational work

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Technical State of Snap 2016
Small number of large monolithic
applications
Projects slowed due to inflexibility
Infrastructure started to be the long pole

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Technical State of Snap 2016 (cont.)
Organizational boundaries also got in
the way
Work was single threaded through central
teams
New product teams were not happy with the
constraints

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Technical State of Snap 2016 (cont.)
Regionalization was impossible in our old
architecture
Performance matters a lot
Stuck with “the way things have always been done”
Teams couldn’t “spin up their service in a new region”

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service-oriented architectures
We had seen the value of microservices in
other organizations
Solve a smaller problem in the best way possible
Separates data, responsibilities (security)
Organizational division
Scaling tied to usage
But what is the best approach to SOA?

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Portability
Our strategy has always been to use the
best of breed
Containers were obvious.
Orchestration is half the battle
Let a vendor do that for us.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes
Amazon EKS
Still highly portable
Less operations – managed
control plane (and more
management coming)
Run it ourselves?
Most portability
But a lot of complexity
And a lot of operational work
Solves a lot of the
problems of managing
a large set of services
+ =

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS at Snap
2018
6 services in
production
today
2019
30-50 services in
production by end
of 2019
End State
Several hundred
services on EKS
Multi-region
Different policies on
redundancy based on
service

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS at Snap
Production services at scale on EKS
7,500 cores 250,000
transactions
per second
High density pod to
node ratio in a secure
service mesh
2019 – Global
regionalization

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EKS Cluster
Standard Architecture
Kubernetes Pod
Envoy
Proxy
Application Service
AuthN/Z
Logs
Metrics

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results
Saved a lot of money
Containers and Amazon EKS give us a lot
of flexibility to adopt new technologies
Envoy is one example, but we expect to continue to
reap this benefit
Performance improvements
Amazon EKS is already widely adopted at
Snap

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next Steps
Continue the march
Service by service
API by API
Optimize regionalization

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap
Kubernetes and AWS
• 51% of Kubernetes workloads run on AWS
• Amazon EKS is Kubernetes Certified
Kubernetes Networking
• The three rules of Kubernetes networking
• CNI plugin
• Kubernetes ServiceTypes
Kubernetes Security
• Pod permissions to an AWS service
Kubernetes Logging
• Implementing logging with EFK

45. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Martin Dominguez
mrtdom@amazon.com
@mp_dominguez