Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Introducing Scality MetalK8s

Introducing Scality MetalK8s, an opinionated Kubernetes distribution with a focus on long-term on-prem deployments, launched by Scality to deploy its Zenko solution in customer datacenters.

It is based on the Kubespray project to reliably install a base Kubernetes cluster, including all dependencies (like etcd), using the Ansible provisioning tool. This installation is further augmented with operational tools for monitoring and metering, including Prometheus, Grafana, ElasticSearch and Kibana. Furthermore, an "ingress controller" is deployed by default, based on Nginx. All of these are managed as Helm packages.

Unlike hosted Kubernetes solutions, where network-attached storage is available and managed by the provider, we assume no such system to be available in environments where MetalK8s is deployed. As such, we focus on managing node-local storage, and exposing these volumes to containers managed in the cluster.

Presented at the OpenStack Summit Vancouver, May 22nd 2018.

Recording available at https://www.youtube.com/watch?v=PzESpAJ6Xgg

https://zenko.io
https://github.com/scality/metal-k8s
https://metal-k8s.readthedocs.io

  • Inicia sesión para ver los comentarios

Introducing Scality MetalK8s

  1. 1. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS Nicolas Trangez - Technical Architect nicolas.trangez@scality.com @eikke
  2. 2. ABOUT SCALITY
  3. 3. ONE PURPOSE GIVING FREEDOM & CONTROL TO PEOPLE WHO CREATE VALUE WITH DATA
  4. 4. 8 60+ 120+ 20+ ~10 GLOBAL CLIENT BASEGLOBAL PRESENCE 20+ OFFICES 200+ PEOPLE NATIONALITIES EUROPEAMERICAS AUSTRALIA JAPAN
  5. 5. OUR JOURNEY TO KUBERNETES Scality RING, S3 Connector & Zenko
  6. 6. Scality RING - Physical servers, some VMs - Only the OS available (incl. ‘Legacy’ like CentOS 6) - Static resource pools - Static server roles / configurations - Solution distributed as RPM packages, deployed using SaltStack - De-facto taking ownership of host, difficult to run multiple instances - Fairly static post-install On-premise Distributed Object & File Storage
  7. 7. Scality S3 Connector On-premise S3-compatible Object Storage - Physical servers, sometimes VMs - Static resource pools - “Microservices” architecture - Solution distributed as Docker container images, deployed using Ansible playbooks - No runtime orchestration - Log management, monitoring,... comes with solution
  8. 8. Scality Zenko - Deployed on-prem or ‘in the Cloud’: major paradigm shift - New challenges, new opportunities - Multi-Cloud Data Controller, must run on multiple Cloud platforms Multi-Cloud Data Controller
  9. 9. Scality Zenko - Embraced Docker as distribution mechanism - Some shared with Scality S3 Connector - For Cloud deployments, started with Docker Swarm - Ran into scaling, reliability and other technical issues - Decided to move to Kubernetes - Managed platforms for Cloud deployments, where available (GKE, AKS, EKS one day) - On-prem clusters Deployment Model
  10. 10. Scality Zenko - Homogenous deployment between in-cloud and on-prem - Various services provided by cluster: - Networking & policies - Service restart, rolling upgrades - Service log capturing & storage - Service monitoring & metering - Load-balancing - TLS termination - Flexible resource management - If needed, easily add resources to cluster by adding some (VM) nodes - HorizontalPodAutoscaler Kubernetes Benefits
  11. 11. OUR JOURNEY TO KUBERNETES MetalK8s
  12. 12. On-prem Kubernetes - Can’t expect a Kubernetes to be available, provided by Scality customer - Looked into various existing offerings, but in the ends needs to be supported by/through Scality (single offering) - Decided to roll our own
  13. 13. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS
  14. 14. OPINIONATED We offer an out-of-the-box experience, no non-trivial choices to be made by users
  15. 15. LONG-TERM Zenko solution is mission-critical, can’t spawn a new cluster to upgrade and use ELB (or similar) in front
  16. 16. ON-PREM Can’t expect anything to be available but (physical) servers with a base OS
  17. 17. Scality MetalK8s - “Stand on the shoulders of giants” - Scope: 5-20 physical machine, pre-provisioned by customer or partner - Built on top of the excellent Kubespray Ansible playbook - Use Kubespray to lay out a base Kubernetes cluster - Also: etcd, CNI - Add static & dynamic inventory validation pre-checks, OS tuning, OS security - Based on experience from large-scale Scality RING deployments - Augment with various services, deployed using Helm - Operations - Ingress - Cluster services - Take care of on-prem specific storage architecture
  18. 18. Scality MetalK8s: Cluster Services - “Stand on the shoulders of giants” - Heapster for dashboard graphs, `kubectl top`,... - metrics-server for HorizontalPodAutoscaler - Looking into k8s-prometheus-adapter - Ingress & TLS termination: nginx-ingress-controller - Cluster monitoring & alerting: Prometheus, prometheus-operator, Alertmanager, kube-prometheus, Grafana - Host-based node_exporter on all servers comprising the cluster, including etcd - Host & container logs: ElasticSearch, Curator, fluentd, Kibana - Considering switch to fluent-bit - All of the above gives a great out-of-the-box experience for operators
  19. 19. Scality MetalK8s: Storage - On-prem: no EBS, no GCP Persistent Disks, no Azure Storage Disk,... - Also: can’t rely on NAS (e.g. through OpenStack Cinder) to be available - Lowest common denominator: local disks in a node - PVs bound to a node, hence PVCs bound, hence Pods bound - Thanks PersistentLocalVolumes & VolumeScheduling! - Decided not to use LocalVolumeProvisioner, but static approach (for now) - Based on LVM2 Logical Volumes for flexibility - PV, VG, LVs defined in inventory, created/formatted/mounted by playbook - K8s PV objects created by playbook - May support whole partitions/drives depending on application need - Working with community on Dynamic Local Volume provisioning - Also using LVM2
  20. 20. Scality MetalK8s: Deployment - Based on years of years of experience deploying Scality RING at enterprise customers, service providers,... - Constraints in datacentra often very different from ‘VMs on EC2’ - No direct internet access: everything through HTTP(S) proxy, no non-HTTP traffic - Dynamic server IP assignment - Security rules requiring services to bind to specific IPs only - Fully air gapped systems: requires 100% offline installation - Non-standard OS/kernel - Integration with corporate authn/authz systems - Not all of the above supported yet, tackling one by one - Relevant patches to be upstreamed to Kubespray - Only support RHEL/CentOS family of Linux distributions - Support for Ubuntu and others can be community-driven, Kubespray supports them - RHEL/CentOS sometimes difficult targets for containers/Docker/Kubernetes
  21. 21. Scality MetalK8s: Ease of Deployment $ # Requirements: a Linux or OSX machine with Python and coreutils-like $ # Create inventory $ vim inventory/... $ make shell # Launches a ‘virtualenv’ with Ansible & deps, ‘kubectl’, ‘helm’ $ # Demo @ https://asciinema.org/a/9kNIpBWg4KiwjT5mNSrH0tmj9 $ ansible-playbook -i inventory -b metal-k8s.yml $ # Grab a coffee, and done
  22. 22. Scality MetalK8s: The road forward - Documentation: Install guides, Operations guides, Troubleshooting guides,... - Forward & backward compatibility requirements - Sizing numbers - Hardware & software compatibility testing - Security auditing & testing - Testing/CI: install, upgrade, downgrade, ‘monkey’,... - Also in very constrained environments - Delivery of fully-offline installation package - ...
  23. 23. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS https://zenko.io https://github.com/scality/metal-k8s @Scality | @Zenko

×