Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

AWS Container Services – 유재석 (AWS 솔루션즈 아키텍트)

588 visualizaciones

Publicado el

본 게시물은 2019년 7월 16일에 AWS Game Team에서 진행한 Container교육 자료입니다. AWS 컨테이너 서비스들을 이해하여 손쉽게 컨테이너를 실행하세요.

Publicado en: Tecnología
  • Sé el primero en comentar

AWS Container Services – 유재석 (AWS 솔루션즈 아키텍트)

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jaeseok Yoo Container, Container, Container …
  2. 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Time 13:00 – 13:15 Docker & Container Orchestration 13:15 – 14:00 Kubernetes & Amazon EKS 14:00 – 14:15 HoL: Launch EKS Cluster 14:15 – 14:30 Break 14:30 – 15:45 HoL: Deploy Dashboard, Microservices, Logging 16:00 – 16:45 Amazon ECS & Dedicated Server on Container 16:45 – 17:00 Clean Up
  3. 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker
  4. 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 애플리케이션의 구성 런 타임 엔진 코드 디펜던시 구성
  5. 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • 다른 애플리케이션 스택 • 다른 하드웨어 배포 환경 • 다른 환경에서 애플리케이션을 실행하는 효율적인 방법은? • 다른 환경으로 쉽게 마이그레이션하는 방법은? 문제점
  6. 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 솔루션 - 도커 이식성 : 이미지 기반 배포 유연성 : 마이크로 서비스 모듈화 신속성 : 가벼운 도커 이미지 효율성 : OS kernel 공유
  7. 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VM과 컨테이너 비교 Server (Host) Host OS Hypervisor App 2 Guest OS Guest OS Guest OS Bins/Libs Bins/Libs Bins/Libs App 1 App 3 VM Server (Host) Host OS Docker Bins/Libs Bins/Libs Bins/Libs App 1 App 2 App 3 Container Hypervisor Guest OS
  8. 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker 이미지 구성 bootfs kernel Base image Image Image W ritable Container add nginx add nodejs U buntu References parent image Base Image : 템플릿으로 사용되는 읽기 전용 이미지 Base Image에서 시작해서 커스텀 Image 추가하는 방식 Dockerfile 활용하여 손쉽게 배포 관련 구성 설정 및 재배포에 용이함
  9. 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile # our base image FROM alpine:3.5 # Install python and pip RUN apk add --update py2-pip # install Python modules needed by the Python app COPY requirements.txt /usr/src/app/ RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt # copy files required for the app to run COPY app.py /usr/src/app/ COPY templates/index.html /usr/src/app/templates/ # tell the port number the container should expose EXPOSE 5000 # run the application CMD ["python", "/usr/src/app/app.py"] $ docker build -t <YOUR_USERNAME>/myfirstapp . Sending build context to Docker daemon 9.728 kB Step 1 : FROM alpine:latest ---> 0d81fc72e790 Step 2 : RUN apk add --update py-pip ---> 976a232ac4ad Removing intermediate container 8abd4091b5f5 Step 3 : COPY requirements.txt /usr/src/app/ ---> 65b4be05340c Step 4 : RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt ---> 8de73b0730c2 Step 5 : COPY app.py /usr/src/app/ … Dockerfile은 컨테이너 내부 이미지 환경 및 구성 정의
  10. 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile best practice - 딱 필요한 Base 파일 선택 From the stock ubuntu image: ubuntu latest 2b1dc137b502 52 seconds ago 458 MB From python:2.7-alpine: alpine latest d3145c9ba1fa 2 minutes ago 86.8 MB
  11. 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 고객사례 - Nextdoor Base OS version Apt packages: OpenSSL libpq syslog-ng Datadog Python runtime PyPI packages: Boto Django Mapnik SendGrid Source code Static assets Images JS CSS
  12. 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Layer 별 각기 다른 업데이트 주기 Quarterly Weekly/ monthly Continuous
  13. 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AMI에서 Docker Container로 변경 Base OS layer System packages Python packages Nextdoor source
  14. 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker 이전에는 빌드 20분 소요 chroot sudo apt-get install sudo pip install git clone make install dpkg create
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Base image , system deps 추가 FROM hub.corp.nextdoor.com/nextdoor/nd_base:precise ADD app/docker/scripts/apt-fast app/docker/scripts/system-deps.sh /deps/ RUN /deps/system-deps.sh
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Python virtualenv 설정 업데이트 ADD app/docker/scripts/venv-deps.sh app/apps/nextdoor/etc/requirements*.txt app/apps/nextdoor/etc/nextdoor.yml app/services/scheduler/etc/scheduler.yml app/services/supervisor/etc/supervisor.yml /deps/ RUN /deps/venv-deps.sh
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. App 소스 업데이트 ADD app/static/nextdoorv2/images /app/static/nextdoorv2/images ADD app/thrift /deps/thrift ADD app/nd /deps/nd ADD app /app
  18. 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 빌드 시간 20분 -> 평균 2분 ECS에 최종 배포까지 평균 5분
  19. 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://docs.docker.com/ https://en.wikipedia.org/wiki/Docker_(software) https://en.wikipedia.org/wiki/LXC https://en.wikipedia.org/wiki/Linux_namespaces https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Chroot https://www.slideshare.net/Docker/creating-effective-images-abby-fuller-aws https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ https://github.com/docker/labs/blob/master/beginner/chapters/webapps.md http://crosbymichael.com/dockerfile-best-practices.html References
  20. 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Common Questions • How do I deploy my containers to hosts? • How do I do zero downtime or blue green deployments? • How do I keep my containers alive? • How can my containers talk to each other? • Linking? Service Discovery? • How can I configure my containers at runtime? • What about secrets? • How do I best optimize my "pool of compute”?
  21. 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How do we make this work at scale?
  22. 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. We need to • start, stop, and monitor lots of containers running on lots of hosts • decide when and where to start or stop containers • control our hosts and monitor their status • manage rollouts of new code (containers) to our hosts • manage how traffic flows to containers and how requests are routed
  23. 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  24. 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance Instance Instance OS OS OS Container Runtime Container Runtime Container Runtime App Service App App Service Service Container Orchestration
  25. 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration myJob: { Cpu: 10 Mem: 256 } Orchestrator Schedule Run “myJob”
  26. 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management OrchestrationService Management §Availability §Lifecycle §Discovery
  27. 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management Orchestration Scheduling §Placement §Scaling §Upgrades §Rollbacks
  28. 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management Orchestration Resource Management § Memory § CPU § Ports
  29. 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are container orchestration tools?
  30. 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Container Services Landscape MANAGEMENT Deployment, Scheduling, Scaling & Management of containerized applications HOSTING Where the containers run Amazon Elastic Container Service Amazon Elastic Container Service for Kubernetes Amazon EC2 AWS Fargate IMAGE REGISTRY Container Image Repository GA : June 6, 2018 Seoul : Jan 11, 2019 Amazon Elastic Container Registry
  31. 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run a (managed) container on AWS AMAZON CONTAINER SERVICES Choose your orchestration tool1 Choose your launch type2 ECS EKS EC2 Fargate EC2 Fargate
  32. 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  33. 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is Kubernetes? Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications
  34. 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Components Control Plane (Controller) Etcd Lightweight, open source Key-Value store containing the cluster API Server Serves the APIs required to manage the cluster Scheduler Determines where (on which nodes) pods will run in the cluster Controller Manager The “worker on the controller” that actually manages the cluster (e.g. replication) Kubernetes Node kubelet Runs the node, starts and stops containers kube-proxy Acts as a network proxy – routes traffic based upon IP and Port. Each service is assigned a unique port on the nodes it runs across, kube-proxy allows that port to be mapped to whatever the service expects. cAdvisor Agent that monitors node health and statistics
  35. 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Architecture
  36. 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Objects • kubectl • Pods • Labels • Deployments • Replication Controllers • Services
  37. 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. kubectl • Command line interface for running commands against the k8s API • Intuitive familiar commands (get, create, describe, delete, etc.) that are simple to learn and easy to use ~/.kube/config k8s master kube-api scheduler
  38. 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pods • A group of one or more containers • Shared: • Data volumes • cgroup • Namespace – network, IPC, etc. node pod1 pod2
  39. 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Labels • Key/Value Pairs • Used to query specific resources within your cluster pod1 pod2 dev prod app001 app001
  40. 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ReplicaSets • Ensure that a specified number of pod “replicas” exist in the cluster 23
  41. 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployments • Declarative updates for Pods and ReplicaSets 23
  42. 42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services • Abstraction which defines a logical set of pods and policy by which to access them
  43. 43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services • Service Discovery: • Environment variables • DNS • Publishing Services: • LoadBalancer (ELB) • ClusterIP, NodePort, External Name (DNS)
  44. 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. kubectl $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-31-24-193.ec2.internal Ready <none> 2m v1.10.3 ip-172-31-36-113.ec2.internal Ready <none> 2m v1.10.3 ip-172-31-65-97.ec2.internal Ready <none> 2m v1.10.3 $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-5blrq 1/1 Running 0 3m kube-system aws-node-btn9b 1/1 Running 0 3m kube-system aws-node-wvd92 1/1 Running 1 3m kube-system kube-dns-64b69465b4-gnzpz 3/3 Running 0 1h kube-system kube-proxy-5prxp 1/1 Running 0 3m kube-system kube-proxy-86q8k 1/1 Running 0 3m kube-system kube-proxy-89stl 1/1 Running 0 3m
  45. 45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dashboard Deploy the dashboard to your cluster $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kub ernetes-dashboard.yaml secret "kubernetes-dashboard-certs" created serviceaccount "kubernetes-dashboard" created role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created deployment.apps "kubernetes-dashboard" created service "kubernetes-dashboard" created Create an eks-admin Account and Cluster Role Binding
  46. 46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dashboard $ vi eks-admin-service-account.yaml apiVersion: v1 kind: ServiceAccount metadata: name: eks-admin namespace: kube-system $ kubectl apply -f eks-admin-service- account.yaml $ eks-admin-cluster-role-binding.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: eks-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: eks-admin namespace: kube-system $ kubectl apply -f eks-admin-cluster-role- binding.yaml
  47. 47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dashboard Retrieve an authentication token $ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') $ kubectl proxy Starting to serve on 127.0.0.1:8001 Access at http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes- dashboard:/proxy/ copy and paste token for login
  48. 48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dashboard
  49. 49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run Nginx $ kubectl run my-nginx --image nginx --port 80 $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE my-nginx 1 1 1 1 13s $ kubectl get pods NAME READY STATUS RESTARTS AGE my-nginx-77f56b88c8-dmvtg 1/1 Running 0 33s $ kubectl describe pod/my-nginx-77f56b88c8-dmvtg Name: my-nginx-77f56b88c8-dmvtg Namespace: default Node: ip-172-31-24-193.ec2.internal/172.31.24.193 Start Time: Fri, 29 Jun 2018 22:04:37 +0900 Labels: pod-template-hash=3391264474 run=my-nginx Annotations: <none> Status: Running IP: 172.31.28.55 Controlled By: ReplicaSet/my-nginx-77f56b88c8
  50. 50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run Nginx - expose within cluster $ kubectl expose deployment my-nginx --target-port=80 [--type=LoadBalancer] $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1h my-nginx ClusterIP 10.100.211.73 <none> 80/TCP 11s $ kubectl edit svc/my-nginx apiVersion: v1 kind: Service … spec: clusterIP: 10.100.211.73 ports: - port: 80 protocol: TCP targetPort: 80 selector: run: my-nginx sessionAffinity: None type: ClusterIP -> LoadBalancer (replace and save) status: loadBalancer: {} add --type=LoadBalancer if you want expose to internet
  51. 51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run Nginx - expose to internet $ watch -n 1 “kubectl get services“ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2h my-nginx LoadBalancer 10.100.211.73 <pending> 80:31743/TCP 7m … NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2h my-nginx LoadBalancer 10.100.211.73 a60e942cbd32d... 80:31743/TCP 7m $ curl http:// a60e942cbd32d11e7992202c08f5229f-284158314.ap-northeast- 2.elb.amazonaws.com * clean up $ kubectl delete svc/my-nginx deployment/my-nginx
  52. 52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run Nginx w/ YAML $ vi my-nginx.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx spec: replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 $ kubectl create -f ./my-nginx.yaml $ kubectl get deployments NAME DESIRED CURRENT UP-TO- DATE AVAILABLE AGE my- nginx 2 2 2 1 6s $ kubectl delete pod my-nginx
  53. 53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run Nginx w/ YAML $ vi my-nginx-app.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: app: nginx spec: type: LoadBalancer ports: - port: 80 selector: app: nginx --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 $ kubectl create -f ./my-nginx-app.yaml $ kubectl get deployments $ kubectl get services
  54. 54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Using Labels $ kubectl label pods -l app=nginx tier=webserver pod "my-nginx-431080787-0fqx9" labeled pod "my-nginx-431080787-d8g3q" labeled pod "my-nginx-431080787-k2r4m" labeled $ kubectl get pods -l app=nginx -L tier NAME READY STATUS RESTARTS AGE TIER my-nginx-431080787-0fqx9 1/1 Running 0 1m webserver my-nginx-431080787-d8g3q 1/1 Running 0 1m webserver my-nginx-431080787-k2r4m 1/1 Running 0 1m webserver
  55. 55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Scaling Application $ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE my-nginx 3 3 3 3 4m $ kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE my-nginx-431080787-0fqx9 1/1 Running 0 4m my-nginx-431080787-d8g3q 1/1 Running 0 4m my-nginx-431080787-k2r4m 1/1 Running 0 4m $ kubectl scale deployment/my-nginx --replicas=2 $ kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE my-nginx-431080787-0fqx9 1/1 Running 0 4m my-nginx-431080787-d8g3q 1/1 Running 0 4m $ kubectl delete -f my-nginx-app.yaml
  56. 56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. and more … In-place updates of resources $ kubectl apply $ kubectl edit $ kubectl patch $ kubectl annotate … Disruptive updates $ kubectl replace $ kubectl rolling-update … $ kubectl autoscale $ kubectl rolling-update … http://kubernetes.io/docs/user-guide/ https://github.com/kubernetes/ku bernetes/tree/master/examples
  57. 57. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run 2048 w/ YAML $ vi my-2048.yaml apiVersion: v1 kind: Service metadata: name: my-2048 labels: app: my-2048 spec: type: LoadBalancer ports: - port: 80 selector: app: my-2048 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-2048 spec: replicas: 1 template: metadata: labels: app: my-2048 spec: containers: - name: my-2048 image: sdscello/2048:1 ports: - containerPort: 80 $ kubectl create -f ./my-2048.yaml $ kubectl get deployments $ kubectl get services * open a browser and connect to the ELB
  58. 58. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run 2048 w/ CI and CD Source Code Github Jenkins Registry Kubernetes Clusterpush trigger build run Enduser ELB
  59. 59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage
  60. 60. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lifecycle of a storage volume Provisioning Binding Using Reclaiming • Static • Dynamic* • Control loop watches for PVC requests and satisfies if PV is available. • For Dynamic, PVC will provision PV • PVC to PV binding is one-to-one mapping • Cluster mounts volume based on PVC • Retain (default) • Recycle • Delete
  61. 61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What if I need specific volume type? StorageClass gp2 io1 sc1 encrypted io1 st1 1) Admin pre-provisions StorageClass based on workload needs 2) End user requests for specific volume types (For ex, encrypted io1 volume) 3) Control loop watches PVC request and allocates volume if PV exists MySQL Pods 4) End user creates stateful workload
  62. 62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage : Storage Class $ vi gp2-storage-class.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: gp2 provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain mountOptions: - debug $ kubectl create -f gp2-storage-class.yaml $ kubectl get storageclass Set gp2 as default storage $ kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is- default-class":"true"}}}’ $ kubectl get storageclass NAME PROVISIONER AGE gp2 (default) kubernetes.io/aws-ebs 24s
  63. 63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage : Persistent Volume $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE * Create 5Gi EBS volume $ aws ec2 create-volume --size 5 --region ap-northeast-2 --availability-zone ap-northeast-2c -- volume-type gp2 { "AvailabilityZone": "us-east-1d", "CreateTime": "2018-07-02T06:29:50.000Z", "Encrypted": false, "Size": 5, "SnapshotId": "", "State": "creating", "VolumeId": "vol-0e9bda6cdc69834a7", "Iops": 100, "Tags": [], "VolumeType": "gp2" } Replace it to your zone
  64. 64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage : Persistent Volume and Claim $ vi my-aws-pv.yaml apiVersion: "v1" kind: "PersistentVolume" metadata: name: "pv0001" spec: capacity: storage: "5Gi" accessModes: - "ReadWriteOnce" awsElasticBlockStore: fsType: "ext4" volumeID: " vol-0e9bda6cdc69834a7" $ kubectl create -f my-aws-pv.yaml $ vi my-aws-pvc.yaml apiVersion: v1 kind : PersistentVolumeClaim metadata: name: pvc0001 spec: storageClassName: "" volumeName: pv0001 accessModes: - ReadWriteOnce resources: requests: storage: 5G $ kubectl create -f my-aws-pvc.yaml
  65. 65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage : Persistent Volume and Claim $ vi my-aws-pvc-pod.yaml apiVersion: v1 kind: Pod metadata: name: redis spec: containers: - name: redis image: redis volumeMounts: - name: pvdemo mountPath: /data volumes: - name: pvdemo persistentVolumeClaim: claimName: pvc0001 $ kubectl create -f my-aws-pvc-pod.yaml $ kubectl describe pods redis Name: redis Namespace: default Node: ip-172-31-36- 113.ec2.internal/172.31.36.113 Start Time: Mon, 02 Jul 2018 17:03:26 +0900 Labels: <none> Annotations: <none> Status: Running IP: 172.31.34.41 Containers: redis: Mounts: /data from pvdemo (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-wtfrw (ro) Volumes: pvdemo: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc0001 ReadOnly: false
  66. 66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage : Persistent Volume * log into the worker instance that is running redis pod $ kubectl get pods NAME READY STATUS RESTARTS AGE Redis 1/1 Running 0 5s $ kubectl exec -it redis -- /bin/bash root@redis:/data# df -h Filesystem Size Used Avail Use% Mounted on overlay 20G 2.8G 18G 14% / tmpfs 998M 0 998M 0% /dev tmpfs 998M 0 998M 0% /sys/fs/cgroup /dev/xvdbw 4.8G 20M 4.6G 1% /data /dev/xvda1 20G 2.8G 18G 14% /etc/hosts shm 64M 0 64M 0% /dev/shm tmpfs 998M 12K 998M 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 998M 0 998M 0% /sys/firmware
  67. 67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
  68. 68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services • A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is (usually) determined by a Label Selector. • Let’s talk about what are the differences between LoadBalancer, NodePort and Ingress
  69. 69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : ClusterIP • Exposes the service on a cluster- internal IP • Only reachable from within the cluster • Access possible via kube-proxy • Useful for debugging services, connecting from your laptop or displaying internal dashboards
  70. 70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : LoadBalancer • Exposes the service externally using a cloud provider’s load balancer. • NodePort and ClusterIP services (to which LB will route) automatically created. • Each service exposed with a LoadBalancer (ELB or NLB) will get its own IP address • Exposes L4 (TCP) or L7 (HTTP) services
  71. 71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : LoadBalancer - Sample $ vi my-nginx-lb.yaml apiVersion: v1 kind: Service metadata: name: my-nginx-lb labels: app: nginx-lb spec: type: LoadBalancer ports: - port: 80 selector: app: nginx-lb --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx-lb spec: replicas: 3 template: metadata: labels: app: nginx-lb spec: containers: - name: nginx-lb image: nginx:1.7.9 ports: - containerPort: 80 $ kubectl create -f ./my-nginx-lb.yaml $ kubectl get deployments $ kubectl get services -o wide // Find ELB name and connect for test * clean up $ kubectl delete –f ./my-nginx-lb.yaml
  72. 72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : LoadBalancer - NLB apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" spec: externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
  73. 73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : NodePort • Exposes the service on each Node’s IP at a static port. • Routes to a ClusterIP service, which is automatically created. • from outside the cluster: <NodeIP>:<NodePort> • 1 service per port • Uses ports 30000-32767
  74. 74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : NodePort - Sample $ vi my-nginx-np.yaml apiVersion: v1 kind: Service metadata: name: my-nginx-np labels: app: nginx-np spec: type: NodePort ports: - port: 80 selector: app: nginx-np --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx-np spec: replicas: 3 template: metadata: labels: app: nginx-np spec: containers: - name: nginx-np image: nginx:1.7.9 ports: - containerPort: 80 $ kubectl create -f ./my-nginx-np.yaml $ kubectl get deployments $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL- IP PORT(S) AGE SELECTOR my-nginx- np NodePort 10.100.90.163 <none> 80:31923/ TCP 4s app=nginx-np
  75. 75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : NodePort - Sample $ kubectl describe services my-nginx-np Name: my-nginx-np Namespace: default Labels: app=nginx-np Annotations: <none> Selector: app=nginx-np Type: NodePort IP: 10.100.90.163 Port: <unset> 80/TCP TargetPort: 80/TCP NodePort: <unset> 31923/TCP Endpoints: 172.31.31.134:80,172.31.41.219:80,172.31.76.169:80 Session Affinity: None External Traffic Policy: Cluster Events: <none> ClusterIP can be accessible from any Pods are running in the cluster 31923 is the port that listen in the workers. You can access the Pod from internet if you open a firewall for workers’ security group
  76. 76. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : NodePort - Sample $ kubectl run -i --tty --image busybox test --restart=Never --rm /bin/sh # wget -qO- 10.100.90.163 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> </html> # exit
  77. 77. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : NodePort - Sample * Update security group to allow the access to the workers from outside of internet * Note Public IP of all the workers and try to connect each of nodes with same port $ curl 54.89.86.193:31923 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> * clean up $ kubectl delete –f ./my-nginx-np.yaml
  78. 78. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : Ingress • Unlike all the above examples, Ingress is actually NOT a type of service. Instead, it sits in front of multiple services and act as a “smart router” or entrypoint into your cluster. • Demo is at the end of the page as it requires helm for ingress controller
  79. 79. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Helm
  80. 80. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Helm from DEIS
  81. 81. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is Helm? • Helm helps you manage Kubernetes applications • Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application. • Charts are easy to create, version, share, and publish • so start using Helm and stop the copy-and-paste madness. https://github.com/kubernetes/helm
  82. 82. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Preparation - helm $ kubectl create serviceaccount --namespace kube-system tiller $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin -- serviceaccount=kube-system:tiller $ helm init --service-account tiller $ kubectl get pods --all-namespaces kube-system tiller-deploy-f5597467b-z6vrm 1/1 Running 0 7m
  83. 83. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ helm search NAME VERSION DESCRIPTION stable/acs-engine-autoscaler 2.1.1 Scales worker nodes within agent pools stable/aerospike 0.1.5 A Helm chart for Aerospike in Kubernetes stable/artifactory 6.2.0 Universal Repository Manager supporting all maj... stable/aws-cluster-autoscaler 0.3.1 Scales worker nodes within autoscaling groups. stable/buildkite 0.2.0 Agent for Buildkite stable/centrifugo 2.0.0 Centrifugo is a real-time messaging server. stable/chaoskube 0.6.0 Chaoskube periodically kills random pods in you... stable/chronograf 0.3.0 Open-source web application written in Go and R... stable/cluster-autoscaler 0.2.1 Scales worker nodes within autoscaling groups. stable/cockroachdb 0.5.1 CockroachDB is a scalable, survivable, strongly... … stable/testlink 0.4.15 Web-based test management system that facilitat... stable/traefik 1.14.2 A Traefik based Kubernetes ingress controller w... stable/uchiwa 0.2.2 Dashboard for the Sensu monitoring framework stable/voyager 2.0.0 Voyager by AppsCode - Secure Ingress Controller... stable/weave-cloud 0.1.2 Weave Cloud is a add-on to Kubernetes which pro... stable/wordpress 0.7.4 Web publishing platform for building blogs and ... stable/zeppelin 1.0.0 Web-based notebook that enables data-driven, in... stable/zetcd 0.1.4 CoreOS zetcd Helm chart for Kubernetes
  84. 84. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ helm install stable/wordpress RESOURCES: ==> v1/Secret NAME TYPE DATA AGE lumpy-mandrill-mariadb Opaque 2 2s lumpy-mandrill-wordpress Opaque 2 2s ==> v1/PersistentVolumeClaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE lumpy-mandrill-mariadb Bound pvc-883cf38a-d348-11e7-9922-02c08f5229fc 8Gi RWO gp2 2s lumpy-mandrill-wordpress Bound pvc-883da980-d348-11e7-9922-02c08f5229fc 10Gi RWO gp2 2s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE lumpy-mandrill-mariadb ClusterIP 10.100.235.4 <none> 3306/TCP 2s lumpy-mandrill-wordpress LoadBalancer 10.100.33.99 a88484869d348... 80:30079/TCP,443:32070/TCP 2s
  85. 85. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ helm install stable/wordpress NOTES: 1. Get the WordPress URL: NOTE: It may take a few minutes for the LoadBalancer IP to be available. Watch the status with: 'kubectl get svc --namespace default -w lumpy-mandrill-wordpress' export SERVICE_IP=$(kubectl get svc --namespace default lumpy-mandrill-wordpress -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo http://$SERVICE_IP/admin 2. Login with the following credentials to see your blog echo Username: user echo Password: $(kubectl get secret --namespace default lumpy-mandrill-wordpress -o jsonpath="{.data.wordpress- password}" | base64 --decode)
  86. 86. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1d zooming-frog-mariadb ClusterIP 10.100.160.162 <none> 3306/TCP 8m zooming-frog-wordpress LoadBalancer 10.100.209.213 a4e9a5ae47c61... 80:32573/TCP,443:32191/TCP 8m $ kubectl describe service lumpy-mandrill-wordpress Name: zooming-frog-wordpress Namespace: default Labels: app=zooming-frog-wordpress chart=wordpress-1.0.9 heritage=Tiller release=zooming-frog Annotations: <none> Selector: app=zooming-frog-wordpress Type: LoadBalancer IP: 10.100.209.213 LoadBalancer Ingress: a4e9a5ae47c6111e8a86112fe8484ed4-1956022530.us-east-1.elb.amazonaws.com Port: http 80/TCP TargetPort: http/TCP NodePort: http 32573/TCP
  87. 87. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ kubectl get secret --namespace default lumpy-mandrill-wordpress –o jsonpath="{.data.wordpress-password}" | base64 –decode * Open a browser and connect to the Wordpress Site and Admin Site
  88. 88. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Wordpress - helm $ ls -al ~/.helm/cache/archive total 64 drwxr-xr-x 4 kimsaeho ANTDomain Users 136 Jun 1 11:58 . drwxr-xr-x 3 kimsaeho ANTDomain Users 102 May 11 17:36 .. -rw-r--r-- 1 kimsaeho ANTDomain Users 15532 Jun 30 21:29 wordpress-1.0.9.tgz $ tar xvfz ~/.helm/cache/archive/wordpress-1.0.6.tgz -C . $ helm ls NAME REVISION UPDATED STATUS CHART NAMESPACE zooming-frog 1 Sat Jun 30 21:30:00 2018 DEPLOYED wordpress-1.0.9 default * clean up $ helm delete --purge zooming-frog look at the some important files (Chart.yaml, values.yaml) that defines how the package is deploying the applications
  89. 89. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : Ingress
  90. 90. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : Ingress • exposes HTTP/HTTPS routes to services within the cluster • Many implementations: ALB, Nginx, F5, HAProxy etc • Default Service Type: ClusterIP
  91. 91. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTP ListenerHTTPS Listener Rule: /cheesesRule: /charcuterie TargetGroup: Green (IP Mode) TargetGroup: Blue (Instance Mode) NodePort NodePort
  92. 92. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample - Extended ELBingress-*.popori.net Nginx Ingress ingress-nginx.popori.net Ingress-tutum.popori.net Jenkins Github Registry build push pull run
  93. 93. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample $ helm install stable/nginx-ingress --name=nginx-ingress --namespace=kube-system --set rbac.create=true NAME: nginx-ingress LAST DEPLOYED: Sun Jul 1 00:35:45 2018 NAMESPACE: kube-system STATUS: DEPLOYED ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress-controller LoadBalancer 10.100.198.62 <pending> 80:30396/TCP,443:30752/TCP 1s nginx-ingress-default-backend ClusterIP 10.100.170.212 <none> 80/TCP 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE nginx-ingress-controller-67b9bf4c56-plhgf 0/1 Running 0 1s nginx-ingress-default-backend-d676cbb5f-xcbzf 0/1 ContainerCreating 0 1s NOTES: The nginx-ingress controller has been installed. It may take a few minutes for the LoadBalancer IP to be available. You can watch the status by running 'kubectl --namespace kube-system get services -o wide -w nginx-ingress- controller'
  94. 94. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample $ vi my-nginx-ingress.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: app: nginx spec: type: ClusterIP ports: - port: 80 selector: app: nginx --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: sdscello/nginx ports: - containerPort: 80 $ kubectl create -f ./my-nginx-ingress.yaml $ kubectl get deployments $ kubectl get services -o wide
  95. 95. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample $ vi my-nginx-ingress-expose.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-nginx-ingress spec: rules: - host: ingress.popori.net http: paths: - path: / backend: serviceName: my-nginx servicePort: 80 * If you don’t have your own domain, you can use ELB DNS Name instead $ kubectl create -f ./my-nginx-ingress- expose.yaml $ kubectl get services -o wide $ kubectl describe services my-nginx-ingress Name: my-nginx-ingress Namespace: default Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- ingress.popori.net / my-nginx:80 (<none>) Annotations: … * Connect to your domain and make sure you can see the nginx index page
  96. 96. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample Let’s run another pod $ vi my-tutum-ingress.yaml apiVersion: v1 kind: Service metadata: name: my-tutum labels: app: tutum spec: type: ClusterIP ports: - port: 80 selector: app: tutum --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-tutum spec: replicas: 3 template: metadata: labels: app: tutum spec: containers: - name: tutum image: tutum/hello-world ports: - containerPort: 80 $ kubectl create -f ./my-tutum-ingress.yaml $ kubectl get deployments $ kubectl get services -o wide
  97. 97. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample $ kubectl edit ingress my-nginx-ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-nginx-ingress namespace: default spec: rules: - host: ingress.popori.net http: paths: - backend: serviceName: my-nginx servicePort: 80 path: / - backend: serviceName: my-tutum servicePort: 80 path: /tutum status: loadBalancer: ingress: - {} Add this lines, so it rewrites /tutum requests to the appropriate pod
  98. 98. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : Ingress - Sample $ curl http://ingress.popori.net <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> … <p><em>Thank you for using nginx.</em></p> </body> </html> $ curl http://ingress.popori.net/tutum <html> <head> <title>Hello world!</title> <link href='http://fonts.googleapis.com/css?family=Open+Sans:4 00,700' rel='stylesheet' type='text/css’> … </head> <body> <img id="logo" src="logo.png" /> <h1>Hello world!</h1> <h3>My hostname is my-tutum-8479747799-8jqks</h3> <h3>Links found</h3> <b>MY_TUTUM</b> listening in 80 available at tcp://10.100.253.39:80<br /> <b>MY_NGINX</b> listening in 80 available at tcp://10.100.50.246:80<br /> <b>KUBERNETES</b> listening in 443 available at tcp://10.100.0.1:443<br /> </body> </html>
  99. 99. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet
  100. 100. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Statefulset Properties • Network identifiers • Persistent Storage • Ordered graceful deployment and scaling • Ordered graceful termination • Ordered rolling updates • If none of these fit your portfolio, use Deployment or Replicaset
  101. 101. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet 1) Define headless service, statefulset and PVC 2) Control loop allocates PV based on PVC request StorageClass gp2 io1 sc1 encrypted io1 st1 3) Kubernetes creates statefulset MySQL Pods mysql-0 mysql-1 mysql-2 mysql-3 Network Identifiers Ordered Deployment
  102. 102. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet 1) Define headless service, statefulset and PVC 2) Control loop allocates PV based on PVC request 3) Kubernetes creates statefulset MySQL Pods mysql-0 mysql-1 mysql-2 mysql-3 Ordered Scaling mysql-4
  103. 103. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet • StatefulSets are intended to be used with stateful applications and distributed systems. • Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.
  104. 104. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ vi my-nginx-ss.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: app: my-nginx spec: ports: - port: 80 clusterIP: None selector: app: my-nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: my-web spec: serviceName: "my-nginx" replicas: 2 selector: matchLabels: app: my-nginx template: metadata: labels: app: my-nginx spec: containers: - name: my-nginx image: nginx:1.7.9 ports: - containerPort: 80 volumeMounts: - name: my-pv mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: my-pv spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi
  105. 105. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ kubectl get pods -w NAME READY STATUS RESTARTS AGE my-web-0 0/1 Pending 0 7s my-web-0 0/1 Pending 0 15s my-web-0 0/1 ContainerCreating 0 15s my-web-0 1/1 Running 0 24s my-web-1 0/1 Pending 0 0s my-web-1 0/1 Pending 0 0s my-web-1 0/1 Pending 0 6s my-web-1 0/1 ContainerCreating 0 6s my-web-1 1/1 Running 0 16s * StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0..N-1}. * Notice that the my-web-1 Pod is not launched until the my-web-0 Pod is Running and Ready
  106. 106. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 6d my-nginx ClusterIP None <none> 80/TCP 1m $ kubectl get statefulset NAME DESIRED CURRENT AGE my-web 2 2 1m $ kubectl get pods NAME READY STATUS RESTARTS AGE my-web-0 1/1 Running 0 2m my-web-1 1/1 Running 0 2m $ kubectl exec -it my-web-0 -- /bin/bash root@my-web-0:/# df -h Filesystem Size Used Avail Use% Mounted on overlay 20G 3.0G 18G 15% / tmpfs 998M 0 998M 0% /dev /dev/xvdbp 976M 2.6M 907M 1% /usr/share/nginx/html
  107. 107. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'hostname'; done my-web-0 my-web-1 $ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'echo $(hostname) > /usr/share/nginx/html/index.html'; done $ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'apt-get -qq update; apt-get -y install curl'; done $ for i in 0 1; do kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done my-web-0 my-web-1
  108. 108. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ kubectl run -i --tty --image busybox test --restart=Never --rm /bin/sh # nslookup my-web-0.my-nginx Server: 10.100.0.10 Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local Name: my-web-0.my-nginx Address 1: 172.31.34.41 my-web-0.my-nginx.default.svc.cluster.local # wget -qO- my-web-0.my-nginx.default.svc.cluster.local my-web-0
  109. 109. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ kubectl get pods -w -l app=my-nginx NAME READY STATUS RESTARTS AGE my-web-0 1/1 Running 0 2d my-web-1 1/1 Running 0 2d my-web-0 1/1 Terminating 0 2d my-web-1 1/1 Terminating 0 2d my-web-0 0/1 Terminating 0 2d my-web-1 0/1 Terminating 0 2d my-web-0 0/1 Terminating 0 2d my-web-0 0/1 Terminating 0 2d my-web-0 0/1 Pending 0 1s my-web-0 0/1 Pending 0 1s my-web-0 0/1 ContainerCreating 0 1s my-web-1 0/1 Terminating 0 2d my-web-1 0/1 Terminating 0 2d my-web-0 1/1 Running 0 11s my-web-1 0/1 Pending 0 1s my-web-1 0/1 Pending 0 1s my-web-1 0/1 ContainerCreating 0 1s my-web-1 1/1 Running 0 1m $ kubectl delete pods -l app=my-nginx pod "my-web-0" deleted pod "my-web-1" deleted
  110. 110. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet - Sample $ for i in 0 1; do ../../kubectl exec -it my-web-$i -- sh -c 'df -h | grep html'; done /dev/xvdbp 976M 2.6M 907M 1% /usr/share/nginx/html /dev/xvdcv 976M 2.6M 907M 1% /usr/share/nginx/html $ for i in 0 1; do ../../kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done sh: 1: curl: not found command terminated with exit code 127 sh: 1: curl: not found command terminated with exit code 127 $ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'apt-get -qq update; apt-get -y install curl'; done $ for i in 0 1; do kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done my-web-0 my-web-1 Because new pods has launched, manually installed curl is no longer exists in the pods. But, the contents (index.html) that stored in EBS volume is still available.
  111. 111. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 57%of Kubernetes workloads run on AWS today — Cloud Native Computing Foundation
  112. 112. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  113. 113. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is Amazon EKS? • Amazon Elastic Container Service for Kubernetes (Amazon EKS) is a managed service that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.
  114. 114. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes on AWS Managed Kubernetes on AWS Highly available Automated version upgrades Integration with other AWS services Etcd Master Managed Kubernetes control plane CloudTrail, CloudWatch, ELB, IAM, VPC, PrivateLink
  115. 115. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes on AWS 3x Kubernetes masters for HA
  116. 116. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Availability Zone 1 Master Master Availability Zone 2 Availability Zone 3 Master Workers Workers Workers Customer Account AWS Managed
  117. 117. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Control Plane
  118. 118. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl
  119. 119. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  120. 120. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture
  121. 121. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CloudWatch Integration
  122. 122. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  123. 123. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Control Plane Highly available and single tenant infrastructure All “native AWS” components Fronted by an NLB VPC API Server ASG Etcd ASG NLB AZ-1 AZ-2 AZ-3 ELB Instances Instances
  124. 124. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Control Plane Master Node Scheduler Controller Manager Cloud Controller Manager API Server etcd
  125. 125. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What happens when I run ‘kubectl create –f pods.yaml’?
  126. 126. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM Authentication Kubectl 3) Authorizes AWS Identity with RBAC K8s API 1) Passes AWS Identity 2) Verifies AWS Identity 4) K8s action allowed/denied AWS Auth
  127. 127. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Control Plane Master Node Scheduler Controller Manager Cloud Controller Manager API Server etcd Kubectl
  128. 128. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Control Plane API Server Kubectl Authorization Webhook RBACaws-iam- authenticator Authentication Admission Controllers Mutating Webhook Validation Webhook
  129. 129. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. kubectl configuration # [...] users: - name: aws user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 command: aws-iam-authenticator args: - "token" - "-i" - "CLUSTER_ID" - "-r" - "ROLE_ARN" # no client certificate/key needed here!
  130. 130. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Authentication and Authorization • User or IAM role who creates EKS cluster gains Admin privileges • This {“super”} user/role can then add additional users or IAM roles and configure RBAC permissions • To add, configure aws-auth Configmap kubectl edit -n kube-system configmap/aws-auth
  131. 131. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. aws-auth configuration apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6 username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::555555555555:user/admin username: admin groups: - system:masters - userarn: arn:aws:iam::555555555555:user/john username: john groups: - pod-admin # k8s RBAC group
  132. 132. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Data Plane
  133. 133. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  134. 134. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Data Plane Worker Node kube-dnsKubelet aws- node Container runtime Control Plane API kube- proxy
  135. 135. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] ExecStartPre=/sbin/iptables -P FORWARD ACCEPT ExecStart=/usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --allow-privileged=true --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS Restart=on-failure RestartForceExitStatus=SIGPIPE RestartSec=5 KillMode=process [Install] WantedBy=multi-user.target
  136. 136. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS AMI Build Scripts https://github.com/awslabs/amazon-eks-ami Source of truth for EKS Optimized AMI Easily build your own EKS AMI Build assets for EKS AMI for each supported Kubernetes version
  137. 137. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Optimized AMI with GPU Support Easily run Tensorflow/Kubeflow on Amazon EKS Includes NVIDIA packages to support Amazon P2 and P3 instances Available on AWS Marketplace
  138. 138. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Worker Node Setup – Bootstrapping /etc/eks/bootstrap.sh <cluster-name> [options] Uses UserData for configuring System resources and extra Kubelet config Reserve compute resources for System Daemons (Kubelet, Container runtime) and Pod eviction thresholds
  139. 139. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Upgrades
  140. 140. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Version Versions supported: 1.10.13, 1.11.8, 1.12.6, 1.13.7 EKS will support up to 3 versions of Kubernetes at once ”Deprecation” will prevent new cluster creation on old version
  141. 141. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Services Roadmap https://github.com/aws/containers-roadmap
  142. 142. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Platform Version Platform Version revisions represent API server configuration changes or Kubernetes patches Platform Versions increment within a Kubernetes version only
  143. 143. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Platform Version
  144. 144. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Kubernetes Version Updates New UpdateClusterVersion API – supports in place updates of Kubernetes version Introduces an ”update” EKS API object ListUpdates and DescribeUpdate APIs to provide visibility into the status of a given update
  145. 145. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Updating Worker Nodes Two options: 1) Create new node group with latest EKS AMI >> taint old nodes >> drain old nodes >> terminate old CFN template 2) Simply update AMI in CFN template; “rolling” replacement policy terminates nodes (Downsides: un-graceful termination of applications)
  146. 146. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Networking & Load Balancing
  147. 147. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS VPC CNI Plugin ENI Secondary IPs: 10.0.0.1 10.0.0.2 10.0.0.1 10.0.0.2 ENI 10.0.0.20 10.0.0.22 Secondary IPs: 10.0.0.20 10.0.0.22 ec2.associateaddress() VPC Subnet – 10.0.0.0/24 Instance 1 Instance 2 VPC
  148. 148. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS VPC CNI plugin – understanding IP allocation Primary CIDR range è RFC 1918 addresses è 10/8, 172.16/12, 192.168/16 Used in EKS for: • Pods • X-account ENIs for (masters à workers) communication (exec, logs, proxy etc.) • Internal Kubernetes services network (10.100/16 or 172.20/16 – chosen based on your VPC range) Setup: • EKS cluster creation è provide list of subnets (in at least 2 AZs!) è tagging
  149. 149. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS VPC CNI plugin – understanding IP allocation Secondary CIDR ranges (new!) è non-RFC 1918 address blocks (100.64.0.0/10 and 198.19.0.0/16) Used in EKS for: • Pods only How? • EKS custom network config è enable è create ENIConfig CRD è annotate nodes CNI 1.2.1+
  150. 150. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Load Balancing All three AWS Elastic Load Balancing products are supported NLB and CLB supported by Kubernetes Service type=LoadBalancer Internal and External Load Balancer support
  151. 151. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Load Balancing Want to use an Internal Load Balancer? Use annotation: service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 Want to use an NLB? Use annotation: service.beta.kubernetes.io/aws-load-balancer-type: nlb
  152. 152. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ALB Ingress Controller Production-Ready 1.0 Release Supported by Amazon EKS Team Open Source Development: https://github.com/kubernetes- sigs/aws-alb-ingress-controller Customers are using it in production today!
  153. 153. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTP ListenerHTTPS Listener Rule: /cheesesRule: /charcuterie TargetGroup: Green (IP Mode) TargetGroup: Blue (Instance Mode) NodePort NodePort Ingress Resource Creation via Kubectl or API
  154. 154. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Windows Support
  155. 155. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CSI Drivers for EFS and FSx Lustre
  156. 156. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS App Mesh GA
  157. 157. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Get Started https://eksworkshop.com Modules: • Health Checks • Logging with Elasticsearch, Fluentd, and Kibana (EFK) • Monitoring using Prometheus and Grafana • Servicemesh with Istio • Stateful Containers using StatefulSets
  158. 158. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  159. 159. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECS ALB ALB AZ 1 AZ 2 user/scheduler Scheduler Cluster State Service Placement Engine Event Stream Internet ECS agent ECS agent ECS agent EC2 instance EC2 instance EC2 instance
  160. 160. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECS EC2 INSTANCES LOAD BALANCER Internet ECS AGENT TASK Container TASK Container ECS AGENT TASK Container TASK Container AGENT COMMUNICATION SERVICE Amazon ECS API CLUSTER MANAGEMENT ENGINE KEY/VALUE STORE ECS AGENT TASK Container TASK Container LOAD BALANCER
  161. 161. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECS : Cluster EC2 INSTANCES LOAD BALANCER Internet ECS AGENT TASK Container TASK Container ECS AGENT TASK Container TASK Container AGENT COMMUNICATION SERVICE Amazon ECS API CLUSTER MANAGEMENT ENGINE KEY/VALUE STORE ECS AGENT TASK Container TASK Container LOAD BALANCER
  162. 162. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECS : Task EC2 INSTANCES LOAD BALANCER Internet ECS AGENT TASK Container TASK Container ECS AGENT TASK Container TASK Container AGENT COMMUNICATION SERVICE Amazon ECS API CLUSTER MANAGEMENT ENGINE KEY/VALUE STORE ECS AGENT TASK Container TASK Container LOAD BALANCER
  163. 163. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tasks are defined via Task Definitions { "containerDefinitions": [ { "name": "simple-app", "image": "httpd:2.4", "cpu": 10, "memory": 300, "portMappings": [ { "hostPort": 80, "containerPort": 80, "protocol": "tcp" } ], "essential": true, "mountPoints": [ { "containerPath": "/usr/local/apache2/htdocs", "sourceVolume": "my-vol" } ] }, { "name": "busybox", "image": "busybox", "cpu": 10, "memory": 200, "volumesFrom": [ { "sourceContainer": "simple-app" } ], "command": [ "/bin/sh -c "..."" ], "essential": false } ], "volumes": [ { "name": “my-vol" } ] }
  164. 164. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tasks are defined via Task Definitions { "containerDefinitions": [ { "name": "simple-app", "image": "httpd:2.4", "cpu": 10, "memory": 300, "portMappings": [ { "hostPort": 80, "containerPort": 80, "protocol": "tcp" } ], "essential": true, "mountPoints": [ { "containerPath": "/usr/local/apache2/htdocs", "sourceVolume": "my-vol" } ] }, 10 CPU units (1024 is a full CPU) 300 MB of memory Expose port 80 in container to port 80 on host Create and mount volumes Essential to our task
  165. 165. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tasks are defined via Task Definitions { "name": "busybox", "image": "busybox", "cpu": 10, "memory": 200, "volumesFrom": [ { "sourceContainer": "simple-app" } ], "command": [ "/bin/sh -c "..."" ], "essential": false } ], "volumes": [ { "name": “my-vol" } ] } From Docker Hub Mount volume from other container Command to exec Volumes
  166. 166. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task log to CloudWatch Logs CloudWatch Logs Amazon S3 Amazon Kinesis AWS Lambda Amazon ElasticSearch Amazon ECS Store Stream Process Search CloudWatch Logs CloudWatch Logs CloudWatch Logs
  167. 167. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM Task Role AWS IAM Amazon DynamoDB S3 AWS IAM DynamoDBRole S3Role Amazon ECS IAM Task Role Identity Access Management (IAM) ECS Task
  168. 168. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task Placement Constraints Name Example AMI ID attribute:ecs.ami-id == ami- eca289fb Availability Zone attribute:ecs.availability- zone == us-east-1a Instance Type attribute:ecs.instance-type == t2.small Distinct Instances type=“distinctInstance” Custom attribute:stack == prod Cluster Constraints Custom Constraints Placement Strategies Apply Filter CPU, memory, port requirements AZ, EC2 type, AMI, or custom constraints Spread or Binpack placement strategy Select final instances for task deployment
  169. 169. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task Placement Strategies Binpacking Spread Affinity Distinct Instance
  170. 170. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example : Instance type and Zone g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small t2.small t2.medium us-east-1aus-east-1d
  171. 171. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECS : Service EC2 INSTANCES LOAD BALANCER Internet ECS AGENT TASK Container TASK Container ECS AGENT TASK Container TASK Container AGENT COMMUNICATION SERVICE Amazon ECS API CLUSTER MANAGEMENT ENGINE KEY/VALUE STORE ECS AGENT TASK Container TASK Container LOAD BALANCER
  172. 172. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task and Service • Split an application that only runs with necessary bin/libs • IAM task role must be set • Restricted use of privileged users within a container • Configure LogDriver (awslogs, fluentd, gelf, journald, splunk, syslog .. ) S e r v i c e sT a s k s • Configure task placement to distribute equally across multiple availability zones • Service Auto Scaling • Application Load Balancer
  173. 173. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CloudWatch ECS Metric 2 Dimensions • ClusterName • ServiceName 4 metrics • CPUReservation • MemoryReservation • CPUUtilization • MemoryUtilization Container Instance … Cluster Task definition Task Service CloudWatch ECS Metrics CloudWatch EC2 Metrics Container Instance Container Instance
  174. 174. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ECS Cluster (EC2 Instance) Auto Scale out Event: Per cluster CPU, memory reservation, or usage New services ECS ECS cluster CloudWatch Developers CloudWatch event
  175. 175. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ECS Cluster (EC2 Instance) Auto Scale in Draining ECS ECS cluster CloudWatch Event: Per cluster CPU, memory reservation, or usage
  176. 176. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service Auto Scaling Amazon EC2 Service Resource buffer (+~15%)
  177. 177. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling Target Tracking Only need to set the target value for the metric (ex: CPU utilization 50%) Auto Scaling automatically adjusts the Task DesiredCount in Service CloudWatch metric ECSServiceAverageCPUUtilization ECSServiceAverageMemoryUtilization ALBRequestCountPerTarget CPUTraffic DesiredCount Time 100% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 5 30 10 15 20 25 Target CPU Utilization DesiredCount
  178. 178. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Get Started https://ecsworkshop.com Modules: • Introduction • Platform • Frontend Rails App • Node.js Backend API • Crystal Backend API
  179. 179. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 게임에서 컨테이너 활용하기
  180. 180. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  181. 181. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Stateless Scale Out Stateful Scale Out Lifetime Stateful Scale Up
  182. 182. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dedicated Server 관리 방식 #1
  183. 183. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dedicated Server 관리 방식 #2 Agent가 대기 명령을 받으면 Git에서 바이너리를 받아와서 실행 execute node.js Agent Dedi Server
  184. 184. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dedicated Server 관리 방식 #2 개선 Agent가 대기 명령을 받으면 도커 레지스트리에서 받아와서 실행 docker run node.js Agent
  185. 185. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container로 운영하기 위한 선결 과제 • 포트 매핑 • static • dynamic • 퍼블릭 주소 알아내기 • 밖에서 알아내기 • 안에서 알아내기
  186. 186. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS 에서 Dedicated Server 운영하기 • Agones Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes • 새로운 pod 타입 GameServer GameServerAllocation • PortPolicy dynamic
  187. 187. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ECS에서 Dedicated Server 운영하기 • Dynamic port mapping for ECS • TaskDefinition에서 host port 영역을 0으로 설정 • Docker 1.6 이후에서 49153 ~ 65535영역을 매핑 • Public IP EC2 Metadata http://169.254.169.254/latest/meta-data/public-ipv4 • Public Port ECS Task Metadata V3: ${ECS_CONTAINER_METADATA_URI}/task V2: http://169.254.170.2/v2/metadata
  188. 188. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  189. 189. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Fargate : Only focus on tasks! Simple, Easy, efficient Serverless Container! =No EC2 Instances to provision, scale or manage ECS Native API , Integrated with VPC, ELB, IAM, CloudWatch and more Pay for CPU, Memory Usage
  190. 190. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Fargate AWS VPC networking mode Advanced task placement Deep integration with AWS platform ECS CLI…{ } Global footprint (in 2018) Powerful scheduling engines Auto scaling CloudWatch metrics Load balancers
  191. 191. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Fargate Scheduling and Orchestration Cluster Manager Placement Engine ECS AMI Docker agent ECS agent EC2 Instance ECS AMI Docker agent ECS agent EC2 Instance ECS AMI Docker agent ECS agent EC2 Instance
  192. 192. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EC2 and AWS Fargate Hybrid cluster ECS Instance ECS Instance ECS Instance ECS InstanceECS Instance ECS Instance EC2 FARGATE Notifications Amazon ECS CLUSTER Availability Zone #1 Availability Zone #2 Availability Zone #3 Subnet 2 172.31.2.0/24 Subnet 1 172.31.1.0/24 Subnet 3 172.31.3.0/24 Web Shopping Cart
  193. 193. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster level isolation PROD Cluster Infrastructure DEV Cluster Infrastructure BETA Cluster Infrastructure QA Cluster Infrastructure Web Web Shopping Cart Shopping Cart Notifications NotificationsWeb Shopping Cart NotificationsWeb Shopping Cart Shopping Cart Notifications NotificationsWeb Web PROD CLUSTER BETA CLUSTER DEV CLUSTER QA CLUSTER
  194. 194. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fargate Define application containers: Image URL, CPU & Memory requirements, etc. register Task Definition create Cluster • Infrastructure Isolation boundary • IAM Permissions boundary run Task • A running instantiation of a task definition • Use Fargate launch type create Service Elastic Load Balancing
  195. 195. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CPU & Memory specification Task Level Resources: • Total CPU/memory across all containers • Required fields • Billing dimensions Units • CPU: cpu-units. 1 vCPU = 1024 cpu-units • Memory: MB Container Level Resources: • Defines sharing of task resources among containers • Optional fields { "family": "scorekeep", "cpu": "1 vCpu", "memory": "2 gb", "containerDefinitions": [ { "name":“scorekeep-frontend", "image":"xxx.dkr.ecr.us-east-1.amazonaws.com/fe“, "cpu": 256, "memoryReservation": 512 }, { "name":“scorekeep-api", "image":"xxx.dkr.ecr.us-east-1.amazonaws.com/api", "cpu": 768, "memoryReservation": 512 } ] } Task Level Resources Container Level Resources Task Definition Snippet
  196. 196. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VPC Integration Launch your Fargate Tasks into subnets Under the hood : • We create an Elastic Network Interface (ENI) • The ENI is allocated a private IP from your subnet • The ENI is attached to your task • Your task now has a private IP from your subnet! You can assign public IPs to your tasks Configure security groups to control inbound & outbound traffic 172.31.0.0/16 Subnet 172.31.1.0/24 Other Entities in VPC EC2 LB DB etc. Private IP 172.31.1.164 ENI Fargate TaskPublic / 208.57.73.13 /
  197. 197. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VPC Configuration { "family": "scorekeep", "cpu": "1 vCpu", "memory": "2 gb", "networkMode": "awsvpc", "containerDefinitions": [ { "name":“scorekeep-frontend", "image":"xxx.dkr.ecr.us-east-1.amazonaws.com/fe", "cpu": 256, "memoryReservation": 512 }, { "name":“scorekeep-api", "image":"xxx.dkr.ecr.us-east-1.amazonaws.com/api", "cpu": 768, "memoryReservation": 512 } ] } $ aws ecs run-task ... -- task-definition scorekeep:1 -- network-configuration “awsvpcConfiguration = { subnets=[subnet1-id, subnet2-id], securityGroups=[sg-id] }” Enables ENI creation & attachment to Task Run Task Task Definition
  198. 198. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Private Task Setup Public subnet Private subnet Fargate TaskENI Private IP 172.31.1.164 NAT Gateway Public EIP 34.214.162.237 Internet Gateway 172.31.0.0/16 172.31.2.0/24 172.31.1.0/24 Destination Target 172.31.0.0/16 local 0.0.0.0/0 NAT Gateway Destination Target 172.31.0.0/16 local 0.0.0.0/0 Internet Gateway Route Tables Internet Attach Internet Gateway to VPC Setup a Public Subnet with • Route to Internet Gateway • NAT Gateway Setup Private Subnet with • Fargate Task • Route to NAT Gateway Security Group to allow outbound traffic Type Port Destination All Traffic ALL 0.0.0.0/0 Outbound Security Group Rules
  199. 199. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Public Task Setup Outbound Inbound Public subnet Fargate Task Public IP 54.191.135.66 Internet Gateway 172.31.0.0/16 172.31.2.0/24 Destination Target 172.31.0.0/16 local 0.0.0.0/0 Internet Gateway Route Table Internet ENI $ aws ecs run-task ... -- network-configuration “awsvpcConfiguration = { subnets=[public-subnet], securityGroups=[sg-id], }” Launch the task into a Public subnet Give it a public IP address Security Group to allow the expected inbound traffic Type Port Source HTTP 8080 0.0.0.0/0 Inbound Security Group Rule Type Port Destination All Traffic ALL 0.0.0.0/0 Outbound Security Group Rules assignPublicIp=ENABLED Run Task
  200. 200. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Internet Facing ELB VPC Setup Public subnet Private subnet Fargate TaskENI Private IP 172.31.1.16 :8080 ALB 172.31.0.0/16 172.31.2.0/24 172.31.1.0/24 Task in private subnet with private IP Task Security GroupALB Security Group Type Port Source HTTP 80 0.0.0.0/0 Inbound Rule Type Port Source Custom TCP 8080 ALB Security Group Inbound Rule Destination Target 172.31.0.0/16 local 0.0.0.0/0 NAT G/W Destination Target 172.31.0.0/16 local 0.0.0.0/0 Internet G/W Public Subnet route table Private Subnet route table ALB in public subnet ALB Security group to allow inbound traffic from internet Task security group to allow inbound traffic from ALB security group
  201. 201. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fargate Storage Layer Storage Space : • 10 GB layer storage available per task across all containers in a single task • Includes image layers Ephemeral storage backed by Amazon EBS Fargate volume Storage : • 4 GB volume space per task • Visible across containers • Configure via task definitions Image Layers Writable Layer Image Layers Writable Layer Container 1 Container 2 10 GB per Task Container 1 Container 2 4 GB Volume Storage mount /var/container1/data /var/container2/data
  202. 202. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PERMISSION TIERS Cluster Permissions Application Permissions Task Housekeeping Permissions Cluster Fargate Task Cluster Permissions: Who can run/see tasks in the cluster? Application (Task) Permissions: Which of my AWS resources can this application access? Housekeeping Permissions: What permissions do I want to grant ECS to perform? e.g. • ECR Image Pull • CloudWatch Logs pushing • ENI creation • Register/Deregister targets into ELB
  203. 203. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. COMPLIANCE 9001/27001/27017/27018
  204. 204. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Fargate Customers ”We don't want to babysit any clusters. That has nothing to do with us” Shimon Tolts CTO, DATREE “We moved to Fargate because we need the ability to scale quickly up from baseline and get fine-grained network control, without having to manage our own infrastructure” Product Hunt
  205. 205. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Entire website runs as microservices. Ruby & GraphQL backend with node.js frontend Needed ability to scale quickly, schedule multi- container workloads, network layer control All in on AWS—Moved entire infrastructure to AWS and Fargate in Jan 2018 Fargate scales quickly with traffic spikes, running multiple services in production Product Hunt: AWS Fargate
  206. 206. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  207. 207. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fargate pricing CPU Memory 256 (.25 vCPU) 512MB, 1GB, 2GB 512 (.5 vCPU) 1GB to 4GB 1024 (1 vCPU) 2GB to 8GB 2048 (2 vCPU) 4GB to 16GB 4096 (4 vCPU) 8GB to 30GB 1 vCPU = $0.04656/hour 1 GB Mem = $0.00511/hour 50 different CPU/memory configurations
  208. 208. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×