Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Writing a Kubernetes Autoscaler with Groovy and Spring Boot

2.719 visualizaciones

Publicado el

Recorded at SpringOne2GX
Presenter: Ray Tsang

Groovy Advanced Track

Join this session to learn about Containers, Kubernetes (and the API). Learn Ray's experience in writing a custom metrics collector plus an autoscaler using Groovy and Spring Boot, deployed as containerized microservices in Kubernetes. We have to go deeper.

Publicado en: Tecnología
  • Sé el primero en comentar

Writing a Kubernetes Autoscaler with Groovy and Spring Boot

  1. 1. @saturnism @kubernetesio @googlecloud Writing a Kubernetes Autoscaler Kuberntes API - In Depth
  2. 2. @saturnism @kubernetesio @googlecloud SPRINGONE2GX WASHINGTON, DC Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Writing a Kubernetes Autoscaler Ray Tsang @saturnism
  3. 3. @saturnism @kubernetesio @googlecloud Ray Tsang Developer Advocate rayt@google.com @saturnism | +RayTsang
  4. 4. @saturnism @kubernetesio @googlecloud Ray Tsang Developer Architect Traveler Photographer flickr.com/saturnism
  5. 5. @saturnism @kubernetesio @googlecloud What we are scaling today? Highly scalable distributed in-memory cache
  6. 6. @saturnism @kubernetesio @googlecloud Distributed in-memory cache Key-value NoSQL store Wonderful Elastic Property Infinispan
  7. 7. @saturnism @kubernetesio @googlecloud Node 1 Cache Node 2 Cache Node 3 Cache A B C
  8. 8. @saturnism @kubernetesio @googlecloud Node 1 Cache Node 2 Cache Node 3 Cache A A B B C C N-Copies for Redundancy
  9. 9. @saturnism @kubernetesio @googlecloud Node 1 Cache Node 2 Cache Node 3 Cache A A B B C C
  10. 10. @saturnism @kubernetesio @googlecloud Node 1 Cache Node 3 Cache A A B C C B
  11. 11. @saturnism @kubernetesio @googlecloud Node 1 Cache A B Node 2 Cache B C Node 3 Cache A Node 4 Cache More Capacity! C
  12. 12. @saturnism @kubernetesio @googlecloud Node 1 Cache A B Node 2 Cache B C Node 3 Cache A Node 4 Cache C Rebalanced!
  13. 13. @saturnism @kubernetesio @googlecloud Wouldn’t it be nice to… Autoscale this?
  14. 14. @saturnism @kubernetesio @googlecloud Horizontal Scaling Vertical Scaling
  15. 15. @saturnism @kubernetesio @googlecloud Stateless Workload Scale Out System Metrics Easy Stuff
  16. 16. @saturnism @kubernetesio @googlecloud Scale In Custom Metrics Stateful Workload Difficult Stuff
  17. 17. @saturnism @kubernetesio @googlecloud Today - Difficult Stuff! Autoscale Infinispan, with target # of entries per node, out and in!
  18. 18. @saturnism @kubernetesio @googlecloud Keep in Mind I’ve never written an autoscaler before!
  19. 19. @saturnism @kubernetesio @googlecloud Oh, By The Way Running in Kubernetes!
  20. 20. @saturnism @kubernetesio @googlecloud Enter Kubernetes Greek for “Helmsman”; also the root of the word “Governor” • Container orchestrator • Runs containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines
  21. 21. @saturnism @kubernetesio @googlecloud Developer View
  22. 22. @saturnism @kubernetesio @googlecloud web browsers Scheduler kubectl web browsers scheduler Kubelet Kubelet Kubelet Kubelet Config file Kubernetes Master Container Image Developer View What just happened?
  23. 23. @saturnism @kubernetesio @googlecloud Kubernetes Exposes REST API
  24. 24. @saturnism @kubernetesio @googlecloud Let’s See It!
  25. 25. @saturnism @kubernetesio @googlecloud Keep in Mind I’ve never written an autoscaler before!
  26. 26. @saturnism @kubernetesio @googlecloud Autoscaling 101 What I learned
  27. 27. @saturnism @kubernetesio @googlecloud Node Your App Metrics Collector JMX Metrics Node Your App Metrics Collector JMX Metrics Metrics Server API Autoscaler Scaling Config Metrics Store Aggregated Metrics Actuator Desired # of Nodes Image / Template Your App Metrics Collector Read Instantiate a new instance Autoscaling Pattern
  28. 28. @saturnism @kubernetesio @googlecloud Autoscaler Configuration Autoscaler Scaling Config Metric Target utilization Image / Template Min # of instances Max # of instances CPU, Memory, QPS, etc. 80% or 100 QPS To start a new instance No less than this many No more than this many
  29. 29. @saturnism @kubernetesio @googlecloud Push Metrics Metric Metric Name Utilization / Value Instance Name Cluster Name Timestamp CPU, Memory, QPS, etc. 80% or 100 QPS node-1431 Infinispan Cluster 12:00:34.123
  30. 30. @saturnism @kubernetesio @googlecloud Instance Name Metric Name Metric Value Timestamp node-1 numberOfEntries 0 23:01:00 node-2 numberOfEntries 0 23:01:00 node-1 numberOfEntries 100 23:01:10 node-2 numberOfEntries 100 23:01:10 node-1 numberOfEntries 150 23:01:20 node-2 numberOfEntries 150 23:01:20 node-1 numberOfEntries 200 23:01:30 node-2 numberOfEntries 200 23:01:30 Example Metrics
  31. 31. @saturnism @kubernetesio @googlecloud Instance Name Metric Name Metric Value Timestamp node-1 numberOfEntries 0 23:01:00 node-2 numberOfEntries 0 23:01:00 node-1 numberOfEntries 100 23:01:10 node-2 numberOfEntries 100 23:01:10 node-1 numberOfEntries 150 23:01:20 node-2 numberOfEntries 150 23:01:20 node-1 numberOfEntries 200 23:01:30 node-2 numberOfEntries 200 23:01:30 Example Metrics
  32. 32. @saturnism @kubernetesio @googlecloud Instance Name Metric Name Metric Value Timestamp node-1 numberOfEntries 0 23:01:00 node-2 numberOfEntries 0 23:01:00 node-1 numberOfEntries 100 23:01:10 node-2 numberOfEntries 100 23:01:10 node-1 numberOfEntries 150 23:01:20 node-2 numberOfEntries 150 23:01:20 node-1 numberOfEntries 200 23:01:30 node-2 numberOfEntries 200 23:01:30 Example Metrics Instance Name Average Metric Value node-1 150 node-2 150 Sum 300
  33. 33. @saturnism @kubernetesio @googlecloud Autoscaler Algorithm target # instances = sum(utilization) target utilization ceil( )
  34. 34. @saturnism @kubernetesio @googlecloud Autoscaler Algorithm sum(300) 100 target # instances = target utilization = 100 ceil( )
  35. 35. @saturnism @kubernetesio @googlecloud Autoscaler Algorithm min < target # instances < max
  36. 36. @saturnism @kubernetesio @googlecloud It’s expensive to scale out Delay the scaling in Watch out for noises
  37. 37. @saturnism @kubernetesio @googlecloud Let’s write an autoscaler! The real stuff
  38. 38. @saturnism @kubernetesio @googlecloud Let’s see some code
  39. 39. @saturnism @kubernetesio @googlecloud I used Spring Boot and Groovy @SpringBootApplication @ComponentScan("org.a8r") @EnableAutoConfiguration @EnableAsync @EnableScheduling class Application { static void main(String[] args) { SpringApplication.run(Application, args) } }
  40. 40. @saturnism @kubernetesio @googlecloud Metrics Store - Used a Tree Cache
  41. 41. @saturnism @kubernetesio @googlecloud Microservices - Yay! @RestController @RequestMapping("/a8r/autoscaler") @Slf4j class AutoscalerService { }
  42. 42. @saturnism @kubernetesio @googlecloud class AutoscalerDefintion { @NotEmpty String replicationControllerId @NotEmpty String metricName @NotNull Double threshold @NotNull Integer duration Integer minReplicas = 1 @NotNull Integer maxReplicas } @RestController @RequestMapping("/a8r/autoscaler") @Slf4j class AutoscalerService { ... }
  43. 43. @saturnism @kubernetesio @googlecloud I Love @Slf4J! @Slf4j class AutoscalerService { … log.info "Updated autoscaler for $fqn, {}", definition … }
  44. 44. @saturnism @kubernetesio @googlecloud Wake up periodically @Scheduled(fixedRateString = "${autoscaler.wakeupInterval}") void autoscale() { autoscalerCache.root.children.each { scale(it.data as AutoscalerDefintion) } } @Async void scale(AutoscalerDefintion definition) { … }
  45. 45. @saturnism @kubernetesio @googlecloud Autoscaler Algorithm
  46. 46. @saturnism @kubernetesio @googlecloud Talking to Kubernetes API
  47. 47. @saturnism @kubernetesio @googlecloud SSLSocketFactory with CA Certificate @Bean SSLSocketFactory sslSocketFactory( @Value("#{environment.KUBERNETES_CA_CERT_FILE}") String caCertFile) throws Exception { ... }
  48. 48. @saturnism @kubernetesio @googlecloud RestTemplate with Headers and SSLSocketFactory restTemplate.setRequestFactory(new SimpleClientHttpRequestFactory() { protected void prepareConnection(HttpURLConnection conn, String httpMethod) throws IOException { if (headers["Authorization"]) { conn.setRequestProperty("Authorization", headers.getFirst("Authorization")) } if (sslSocketFactory && conn instanceof HttpsURLConnection) { ((HttpsURLConnection) conn).setSSLSocketFactory(sslSocketFactory) } super.prepareConnection(conn, httpMethod); };
  49. 49. @saturnism @kubernetesio @googlecloud Certificate Authority… Ugh! Let’s see the code
  50. 50. @saturnism @kubernetesio @googlecloud http://github.com/jmxtrans JMXTrans to the rescue!
  51. 51. @saturnism @kubernetesio @googlecloud { "servers" : [ { "url": "service:jmx:http-remoting-jmx://${infinispan.host}:${infinispan.port}", "queries" : [ { "obj": "jboss.infinispan:type=Cache,component=Statistics,name="namedCache(dist_sync)",*", "attr": ["numberOfEntries"], "outputWriters": [ { "@class": "com.googlecode.jmxtrans.model.output.A8RWriter", "metricName": "custom.cloudmonitoring.googleapis.com/infinispan/namedCache/numberOfEntries" } ] } ] } ] } JMX → Output Writer
  52. 52. @saturnism @kubernetesio @googlecloud Live Demo! Demo time
  53. 53. @saturnism @kubernetesio @googlecloud Try Kubernetes Today!
  54. 54. @saturnism @kubernetesio @googlecloud In the Roadmap - Native Autoscaling in Kubernetes Nodes Pods Horizontal # of nodes # of pods Vertical resources for a node resources for a pod
  55. 55. @saturnism @kubernetesio @googlecloud Horizontal Pod Autoscaling https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/autoscaling.md Autoscaler is First Class Citizen Autoscaler Controller and Resource Using Pod CPU / Memory Utilization
  56. 56. @saturnism @kubernetesio @googlecloud Vertical Pod Autoscaling https://github.com/kubernetes/kubernetes/issues/10782 “vertical auto-sizer sets the compute resource limits and request for pods which do not have them set, and periodically adjust them based on demand signals”
  57. 57. @saturnism @kubernetesio @googlecloud Horizontal Node Autoscaling https://github.com/kubernetes/kubernetes/issues/11748 Pretty much done on Google Compute Engine! Using Node CPU / Memory Utilization
  58. 58. @saturnism @kubernetesio @googlecloud Try out Google Container Engine https://cloud.google.com/container-engine/
  59. 59. @saturnism @kubernetesio @googlecloud Thanks! Images by Connie Zhou http://kubernetes.iohttp://bit.ly/1QLg5E1

×