Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Monitoring kubernetes across data center and cloud

1.706 visualizaciones

Publicado el

Specifically Monitoring Tectonic and Google Container Engine using Datadog.

Publicado en: Tecnología
  • Inicia sesión para ver los comentarios

Monitoring kubernetes across data center and cloud

  1. 1. Monitoring Kubernetes Across Data Center and Cloud Specifically Tectonic and Google Container Engine using Datadog Presenters: Ilan Rabinovitch, Director of Technical Community, Datadog Aleks Saul, Customer-Facing Engineer, CoreOS Aparna Sinha, Senior Product Manager, Google
  2. 2. Google Cloud Platform Kubernetes at a glance Open source production-grade container scheduling and management ● Top 0.01% of all GitHub projects: 950+ contributors & 35,000+ commits Run Anywhere: multi-cloud, on-prem, bare-metal, OpenStack etc Broad industry adoption Commercial Enterprise Support Kubernetes at a glance
  3. 3. Google Cloud Platform Kubernetes provides container-centric infrastructure Once specific containers are no longer bound to specific machines/VMs, host-centric infrastructure no longer works • Scheduling: Decide where my containers should run • Lifecycle and health: Keep my containers running despite failures • Scaling: Make sets of containers bigger or smaller • Naming and discovery: Find where my containers are now • Load balancing: Distribute traffic across a set of containers • Storage volumes: Provide data to containers • Logging and monitoring: Track what’s happening with my containers • Debugging and introspection: Enter or attach to containers • Identity and authorization: Control who can do things to my containers
  4. 4. Google Cloud Platform Kubernetes offers choice and flexibility for Hybrid Cloud Setting up and managing a cluster • Choose a cloud: GCP, AWS, Azure, Rackspace, on-premises, ... • Choose a node OS: CoreOS, Atomic, RHEL, Debian, CentOS, Ubuntu, ... • Provision machines: create VMs, install Docker, ... • Configure networking: IP ranges for Pods, Services, SDN, firewalls, ... • Start cluster services: DNS, logging, monitoring, … • Start and configure Kubernetes • Manage nodes: kernel upgrades, OS updates, hardware failures, … GKE is Google hosted and managed Kubernetes • Directly uses upstream open source • Rolls out within 3-5 business days of the latest open source release • Alpha features also now available through ‘alpha clusters’
  5. 5. Google Cloud Platform Google Container Engine (GKE) “It delivers a high-performing, flexible infrastructure that lets us independently scale components for maximum efficiency” ~ Philips (Hue Lights) “Made our engineers more productive and helped us do more work with less staff” ~ CCP Games (EVE Online)
  6. 6. Google Cloud Platform How Monitoring Works in Google Container Engine Master Storage BackendHeapster Kubelet cAdvisor Node Kubelet cAdvisor Node
  7. 7. Google Cloud Platform Google Container Engine Monitoring Server Metrics used for self repair, and exposed to end users via Stackdriver Primary job is to ensure that each Kubernetes master is available ● Implements the repair logic for when a cluster is non-responsive ● Automatically resizes master machines as the number of nodes grows Also collects metrics for each cluster ● Number of resources (nodes, pods, services, namespaces, etc) ● CPU usage, limit, utilization ratio; Memory usage and limit; Page faults; Disk usage and limit; Uptime ● Uses number of nodes for report billing status
  8. 8. Google Cloud Platform Pluggable interface for cloud monitoring Run Influx and Grafana in the cluster ● alternative to Google Cloud Monitoring Plug in your own! ● e.g., Prometheus, Datadog etc. Kube State metrics: (node status, node capacity, replica state, etc) Prometheus
  9. 9. Google Cloud Platform Kube State Metrics ● Generates metrics about the state of Kubernetes logical objects (node status, node capacity, replica state, etc) ● Deployed alongside your other applications as a kubernetes service. ● Exposes metrics via HTTP API or Prometheus format
  10. 10. Google Cloud Platform We focus on delivering the capabilities required by enterprise organizations to run and manage kubernetes at scale... ● Cluster installers (for AWS and bare metal, to start). ● Management software to upgrade, backup, rollback, scale up and down the cluster. ● Console UI that surfaces management functionality, cluster information, and compute usage to the user and includes add on services (Quay, identity and authentication). Extending Kubernetes for the Enterprise
  11. 11. Google Cloud Platform Tectonic Extends Upstream Kubernetes ● Container orchestration ● Horizontal scale ● High availability ● Service discovery & load balancer ● Installer ● Management console ● Painless updates ● Cluster scaling ● Disaster recovery ● Alerts and logging ● Security (integrated) ● Container registry (Quay) ● Integration across environments Extending Kubernetes for the Enterprise Security Mgmt Kubernetes CoreOS Linux Cloud Integration Container Registry Storage & Compute apps/container/microservices
  12. 12. Google Cloud Platform Tectonic Kubernetes Security ● Clair: container vulnerability scanning ● KMS integration ● LDAP integration ● RBAC integration Extending Kubernetes for the Enterprise Mgmt Kubernetes CoreOS Linux Cloud Integration Container Registry Storage & Compute apps/container/microservices Security
  13. 13. •SaaS based infrastructure and application monitoring •Focus on modern environments •Cloud, Containers, Microservices •Dynamic configuration models •Processing nearly a trillion data points per day •Intelligent Alerting and Insightful Dashboards •Anomaly and Outlier Detection Datadog Overview
  14. 14. Collecting data is cheap; not having it when you need it can be expensive
  15. 15. Operating Systems, Cloud Providers, Containers, Web Servers, Datastores, Caches, Queues and more... Monitor Everything
  16. 16. Datadog ● Deployed as a DaemonSet. One instance per node. ● Collects metrics and events from: ○ container engine (eg Docker) ○ Kubernetes Heapster ○ kube-state-metrics ○ Deployed Applications ○ Google Monitoring APIs ● Exposes statsd end point for custom metrics. ● Metrics are automatically tagged by PODs, Labels, etc
  17. 17. Operational Complexity Increases with.. • Number of things to measure • Velocity of change
  18. 18. How much we measure? 1 instance • 10 metrics from cloud providers 1 operating system (e.g., Linux) • 100 metrics 50~ metrics per application
  19. 19. Operational Complexity 100 instances 500 containers
  20. 20. Operational Complexity: Scale 160 metrics per host 800 metrics per host Assuming 5 containers per host
  21. 21. Operational Complexity: Scale 100 instances 80,000 metrics Assuming 5 containers per host
  22. 22. How much we measure? 1 instance • 10 metrics from cloud providers 1 operating system (e.g., Linux) • 100 metrics 50~ metrics per application N containers • 150*N metrics Metrics Overload!
  23. 23. Operational Complexity Increases with.. • Number of things to measure • Velocity of change
  24. 24. Source: Datadog
  25. 25. Operational Complexity Increases with.. • Number of things to measure • Velocity of change
  26. 26. Monitoring Questions • Where is a given container running? • What is the overall capacity of my cluster? • What port(s) are my applications running on? • What’s the total throughput of my application? • What’s its response time per tag? (app, version, data center) • What’s the distribution of 5xx error per container? What about by data center?
  27. 27. Host Centric
  28. 28. Service Centric
  29. 29. Query Based Monitoring “What’s the average throughput of application:nginx per version ?” “Alert me when one of my pod from replication controller:foo is not behaving like the others?” “Show me rate of HTTP 500 responses from nginx” “… grouped by data center … running my app version 2….”
  30. 30. Service Discovery Docker API Kubernetes Monitoring Agent Container A O A O Containers List & Metadata Additional Metadata (Tags, etc) Config Backends Integration Configurations Host Level Metrics
  31. 31. Q&A You can also follow us on Twitter: @datadoghq @googlecloud @tectonicstack

×