Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
How to integrateHow to integr...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Brace for impact!
What are co...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
This guy
… is called Claudio ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
What are containers?
?
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
20 feet standard container (1...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
They stack up!
By stacking co...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
In a perfect world
The curren...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
The world is not perfect
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
When we talk containers, do w...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Containers are believed to be...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Linux Containers
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Linux Containers
- Can ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
root@container:~# ps auxf
USE...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
root@container:~# df ­h ­x cg...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
root@host:~# free ­m
        ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Above: Host
Below: Container
...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Monitoring memory (LXCF...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
root@host:~# free ­m
        ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Monitoring CPU usage
- ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Using check_lxc
- check...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – check_lxc in Icinga2
- ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – check_lxc in Icinga2
- ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
LXC – Recap
- Can “mostly” be...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Application Containers
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Application Containers (AC)
-...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Monitoring challenges
- ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Management w/ Rancher
- ...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Management w/ Rancher
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Healthchecks! (Kubernete...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Healthcheck in Rancher 2
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Healthcheck in kubectl
$...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Healthcheck in Rancher A...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – Using check_rancher2
- c...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – check_rancher2 example
-...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
AC – check_rancher2 example
$...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Application Containers – Reca...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
References and links
- Lost a...
It’s all about the... containers!
Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler
Thank you
[[ $questions ­eq 0...
Próxima SlideShare
Cargando en…5
×

Its all about the... containers!

29 visualizaciones

Publicado el

How to integrate modern containers into a classical system monitoring. This covers both LXC (System Containers) and Docker/Kubernetes (Application) containers.
It starts with a brief introduction into the world of containers and then uses two examples (check_lxc and check_rancher2) how to monitor the two types of containers.

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Its all about the... containers!

  1. 1. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler How to integrateHow to integrate modern containersmodern containers in a classicalin a classical system monitoringsystem monitoring
  2. 2. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Brace for impact! What are containers? - A real world comparison - Brief introduction? LXC LXC – System Containers - Short Introduction - Monitoring System Containers - check_lxc (Application) Containers - Quick Introduction - Monitoring Challenges - Rancher (2) - check_rancher2
  3. 3. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler This guy … is called Claudio Kuenzler … lives in Switzerland … reports to a master process, co-managing two forks … works at NZZ Media Group and co-founded Infiniroot.com … is @Napsty on Github and @ClaudioKuenzler on Twitter … runs a blog at claudiokuenzler.com … started using Nagios® in 2006, discovered #monitoringlove … maintains several monitoring plugins, best known are: check_esxi_hardware, check_smart, check_equallogic … been using containers since 2012
  4. 4. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler What are containers? ?
  5. 5. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler 20 feet standard container (1 TEU) ~ 20 feet long (6.1m) ~ 8 feet wide (2.4m) ~ 8 feet high (2.5m) ~ 33 m3 volume 40 feet standard container (= 2 TEU) ~ 40 feet long (12.2m) ~ 8 feet wide (2.4m) ~ 8 feet high (2.5m) ~ 67 m3 volume TEU = Twenty feet equivalent unit That's a container!
  6. 6. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler They stack up! By stacking containers together: - Efficiency (use of space) - Stability (they don't wobble around) - Security (for on board staff)
  7. 7. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler In a perfect world The current largest container vessel, the “OOCL Hong Kong”, holds a capacity of 21’413 TEU.
  8. 8. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler The world is not perfect
  9. 9. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler When we talk containers, do we mean…? - Linux Containers (LXC) – aka System Containers → Lightweight VM - (Docker) Containers – aka Application Containers → Single process (mostly) Virtualization on process level - The hard truth: a container is a process - Processes started inside a container are (obviously) child processes - Containers use the same Kernel as the host (process, remember?) - Direct hardware access through Kernel (no hardware virtualization) - Resource allocation/limits using cgroups Again… What are containers?
  10. 10. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Containers are believed to be “new” but are actually “old” in the Unix world - FreeBSD “Jails” exist since FreeBSD 4.0 (2000) - Solaris “Zones” exist since Solaris 10 (2004) - OpenVZ “Containers” exist since 2005 - Linux Containers (LXC) exist since 2007 First developped at IBM, now maintained by Canonical (Ubuntu) AKA System Containers (to distinguish from Docker containers) - Docker Containers (based on liblxc) exist since 2013 Since 2014 with own library (libcontainer) AKA Application Containers - containerd container runtime (since 2015) of Docker Inc. In 2017 Docker Inc donated containerd to CNCF A brief history...
  11. 11. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Linux Containers
  12. 12. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Linux Containers - Can be compared to a classical virtual machine w/o hardware virtualization - Dedicated virtual nic (bridged veth by default), full network access - Dedicated file system (rootfs, best practice: LVM LV) - Dedicated namespaces for isolation - cannot see processes of the host - neither of other containers on the same hosts - Dedicated init system - Basically: A super fast VM! (Fast creation, fast boot) - Install monitoring agents/daemons as you would do on a VM
  13. 13. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler root@container:~# ps auxf USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND root     27207  0.1  0.0  20068  4036 ?        Ss   21:21   0:00 /bin/bash root     27274  0.0  0.0  38308  3348 ?        R+   21:21   0:00  _ ps auxf root         1  0.0  0.0 204336  6660 ?        Ss   Sep03   0:09 /sbin/init root        16  0.0  0.2 374092 267440 ?       Ss   Sep03   1:53 /lib/systemd/systemd­ journald root        35  0.0  0.0  29664  2820 ?        Ss   Sep03   0:03 /usr/sbin/cron ­f root        36  0.0  0.0 250116  3492 ?        Ssl  Sep03   3:21 /usr/sbin/rsyslogd ­n nagios      71  0.0  0.0  23916  4196 ?        Ss   Sep03   0:03 /usr/sbin/nrpe ­c  /etc/nagios/nrpe.cfg ­f root        75  0.0  0.0  12668  1644 pts/3    Ss+  Sep03   0:00 /sbin/agetty ­­noclear  tty4 linux root        77  0.0  0.0  12668  1560 pts/0    Ss+  Sep03   0:00 /sbin/agetty ­­noclear  tty1 linux root        80  0.0  0.0  12668  1644 pts/2    Ss+  Sep03   0:00 /sbin/agetty ­­noclear  tty3 linux root        81  0.0  0.0  12668  1656 pts/1    Ss+  Sep03   0:00 /sbin/agetty ­­noclear  [...] LXC – Monitoring processes → Use check_procs !
  14. 14. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler root@container:~# df ­h ­x cgroup Filesystem             Type         Size  Used Avail Use% Mounted on /dev/vgdata/irczsrvc03 ext4          25G   17G  6.5G  73% / none                   tmpfs        492K     0  492K   0% /dev proc                   proc            0     0     0    ­ /proc proc                   proc            0     0     0    ­ /proc/sys/net proc                   proc            0     0     0    ­ /proc/sys proc                   proc            0     0     0    ­ /proc/sysrq­trigger sysfs                  sysfs           0     0     0    ­ /sys sysfs                  sysfs           0     0     0    ­ /sys sysfs                  sysfs           0     0     0    ­ /sys/devices/virtual/net sysfs                  sysfs           0     0     0    ­ /sys/devices/virtual/net fusectl                fusectl         0     0     0    ­ /sys/fs/fuse/connections devpts                 devpts          0     0     0    ­ /dev/console devpts                 devpts          0     0     0    ­ /dev/pts devpts                 devpts          0     0     0    ­ /dev/tty1 devpts                 devpts          0     0     0    ­ /dev/tty2 devpts                 devpts          0     0     0    ­ /dev/tty3 devpts                 devpts          0     0     0    ­ /dev/tty4 tmpfs                  tmpfs         60G     0   60G   0% /dev/shm tmpfs                  tmpfs         60G  169M   59G   1% /run tmpfs                  tmpfs        5.0M     0  5.0M   0% /run/lock tmpfs                  tmpfs         60G     0   60G   0% /sys/fs/cgroup mqueue                 mqueue          0     0     0    ­ /dev/mqueue binfmt_misc            binfmt_misc     0     0     0    ­ /proc/sys/fs/binfmt_misc hugetlbfs              hugetlbfs       0     0     0    ­ /dev/hugepages LXC – Monitoring filesystem(s) → Use check_disk !
  15. 15. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler root@host:~# free ­m               total        used        free      shared  buff/cache   available Mem:         120869        7296       46183        2839       67390      109749 Swap:         15258           0       15258 root@container:~# free ­m                total        used        free      shared  buff/cache   available Mem:         120869        7296       46182        2839       67390      109748 Swap:         15258           0       15258 LXC – Monitoring memory That’s the same! - Container sees total capacity and used memory of host - Not able to determine own memory usage within container - Therefore do not use monitoring plugin inside container (e.g. check_mem)
  16. 16. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Above: Host Below: Container LXC – Monitoring memory - Same memory usage - Same CPU load - Same uptime - Tasks (procs) differ
  17. 17. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Monitoring memory (LXCFS) Above: Host Below: Container - Still same CPU load - But memory usage differs - Uptime now differs, too - Tasks (procs) differ - lxcfs virtualizes parts of /proc inside the container - lxcfs package in Ubuntu is recommended when installing liblxc1 (since LXC 2.x) - In Debian, needs to be installed manually
  18. 18. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler root@host:~# free ­m               total        used        free      shared  buff/cache   available Mem:           3945         229        2894           0         821        3470 Swap:          3814           0        3814 root@container:~# free ­m                total        used        free      shared  buff/cache   available Mem:           3945          93        3663           0         187        3851 Swap:          3814           0        3814 LXC – Monitoring memory (LXCFS) Used is not the same anymore! - Able to show own memory usage inside container - Container still sees total memory capacity of host - However: Misleading “available” calculation because not all consumers seen - If you run a memory check within the container, use “used” column only That’s wrong! X
  19. 19. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Monitoring CPU usage - Container will always see host usage - Currently not possible to have a “different” view inside container - (Clumsy) Approach: Compare jiffies on the host - Example for time spent on system CPU (kernel space) in 5s: 12215974 – 12215646 = 328 - Compared with the host’s jiffies, gives an idea about the container usage - Future: Possible to monitor CPU usage inside container with cgroups → soon root@host:~# lxc­cgroup ­n container cpuacct.stat user 41618658 system 12215646 root@host:~# sleep 5 root@host:~# lxc­cgroup ­n container cpuacct.stat user 41619791 system 12215974
  20. 20. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Using check_lxc - check_lxc is a monitoring (workaround) plugin executed on the LXC host - Uses cgroup values for memory and cpu checks (lxc-cgroup) - Checks container autostart configuration - Correctly monitors memory usage of container(s), incl. Swap - Gives an idea about container cpu usage root@host:~# /usr/lib/nagios/plugins/check_lxc.sh ­n container01 ­t auto LXC AUTOSTART OK root@host:~# /usr/lib/nagios/plugins/check_lxc.sh ­n container01 ­t mem  LXC container01 OK ­ Used Memory: 571 MB|mem=598769664B;0;0;0;0 root@host:~# /usr/lib/nagios/plugins/check_lxc.sh ­n container01 ­t cpu LXC container01 OK ­ CPU Usage: 27%|cpu=27%;;;0;0 root@host:~# /usr/lib/nagios/plugins/check_lxc.sh ­n container99 ­t cpu LXC container99 OK ­ CPU Usage: 3%|cpu=3%;;;0;0
  21. 21. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – check_lxc in Icinga2 - Deploy checks of LXC containers with “apply” rules - Example: Define a custom variable “containers” on the host object: - Apply rule (here used with nrpe remote check): object Host "lxchost01" {   import "generic­host"   address = "192.168.100.101" […]   # Containers running on this host   vars.containers = [ "container01", "container02", "container99" ] } apply Service "LXC Memory " for (container in host.vars.containers) {   import "generic­service"   check_command = "nrpe"   vars.nrpe_command = "check_lxc"   vars.nrpe_arguments = [ container, "mem" ]   assign where host.address && host.vars.containers }
  22. 22. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – check_lxc in Icinga2 - All (defined) containers are monitored - Not much config changes needed - Quick overview which container uses → most memory → most cpu - check_lxc is still in development → contributions welcome
  23. 23. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler LXC – Recap - Can “mostly” be monitored the same way as a classical host or VM - Some resources must be monitored from “outside” (= on the LXC host) - CPU resource monitoring might soon work from “inside”, too! Monitoring... Where? Example Plugin Processes Inside check_procs Filesystem Inside check_disk Network IO Inside check_netio Memory usage Outside check_lxc CPU usage Outside check_lxc Disk IO Outside check_diskio
  24. 24. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Application Containers
  25. 25. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Application Containers (AC) - Single process application running in a container (supposed to be) - Stateless (no data stored inside container) - Dedicated virtual nic (veth by default), NATted network access (expose) - Dedicated file system (aufs or overlayfs by default, share the capacity) - Dedicated namespaces for isolation - cannot see processes of the host - neither of other containers on the same hosts - No init system – Just a process to start, remember? - Great for quickly scaling up redundant applications (behind a LB) - We rarely hear “Docker” anymore these days. Kubernetes! containerd!
  26. 26. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Monitoring challenges - No additional software/daemons “allowed” (single process, remember?) - No direct network access (NAT via host bridge, iptables) - Expose ports? Yes, but workaroundish and legacy (→ Ingress) - Standalone Docker means a lot of manual work (→ use Orchestration) - Stop treating application containers as a classical host/network object - Think of it as a process, not a host
  27. 27. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Management w/ Rancher - Rancher is a “management layer” on top of the orchestration layer → Container Runtime → Orchestration (e.g. Kubernetes) → Rancher - Intuitive user interface and a flexible HTTP API (→ CI/CD!) - Rancher was chosen after >1y of internal Docker research, comparisons, tests - Since Q3 2017 in production with total >1200 containers (Oct 18) - Rancher 1.x uses Cattle orchestration - Rancher 2.x is relatively new (April 2018), built on Kubernetes orchestration - Rancher 2.x Working on Test + Staging environments, soon first Prod env - Need to monitor the Rancher 2.x environments! PS: No, I’m not affiliated with Rancher! Just a community user.
  28. 28. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Management w/ Rancher
  29. 29. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Healthchecks! (Kubernetes) - The container (pod) can be monitored using readiness and liveness probes - readinessProbe: Detect when the application is ready (e.g. startup delay) - livenessProbe: Detect failures in application (e.g. http error) - Allows multiple kind of probes: - Run a command (e.g. cat /tmp/healthy) → exit 0 = OK - HTTP Check (e.g. GET /health on port 8080) → Status 2xx/3xx = OK - TCP Check (e.g. establish connection to port 8080) → Established = OK
  30. 30. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Healthcheck in Rancher 2
  31. 31. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Healthcheck in kubectl $ kubectl describe pod nginxtest­pfbwm ­­namespace gamma ­­insecure­skip­tls­verify=true Name:           nginxtest­pfbwm Namespace:      gamma Node:           mh­radoi02­t/192.168.254.62 Start Time:     Tue, 02 Oct 2018 21:16:09 +0200 Labels:         controller­revision­hash=3146803588                 pod­template­generation=4                 workload.user.cattle.io/workloadselector=daemonSet­gamma­nginxtest Annotations:    cni.projectcalico.org/podIP: 10.42.1.118/32 Status:         Running IP:             10.42.1.118 Controlled By:  DaemonSet/nginxtest Containers:   nginxtest:     Container ID:   docker://93c32c0f3eaf34f939347206c5e7151eac60efb14ef4a464fb3c82fa5cbde659     Image:          nginx     Image ID:       docker­ pullable://nginx@sha256:e8ab8d42e0c34c104ac60b43ba60b19af08e19a0e6d50396bdfd4cef0347ba83     Port:           <none>     Host Port:      <none>     State:          Running       Started:      Tue, 02 Oct 2018 21:16:12 +0200     Ready:          True     Restart Count:  0     Liveness:       http­get http://:80/ delay=10s timeout=2s period=2s #success=1 #failure=3     Readiness:      http­get http://:80/ delay=10s timeout=2s period=2s #success=2 #failure=3     Environment:    <none>     Mounts:       /var/run/secrets/kubernetes.io/serviceaccount from default­token­9nbfr (ro) Conditions:   Type           Status   Initialized    True    Ready          True    PodScheduled   True  [...]
  32. 32. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Healthcheck in Rancher API $ curl ­s ­u token­xxxxx:longsecretpass https://rancher2.example.com/v3/project/c­r8ss7:p­ 85rmm/pods/gamma:nginxtest­pfbwm |jshon |more { [...]    "name": "nginxtest",    "privileged": false,    "readOnly": false,    "readinessProbe": {     "failureThreshold": 3,     "initialDelaySeconds": 10,     "path": "/",     "periodSeconds": 2,     "port": 80,     "scheme": "HTTP",     "successThreshold": 2,     "tcp": false,     "timeoutSeconds": 2,     "type": "/v3/project/schemas/probe"    },    "resources": {     "type": "/v3/project/schemas/resourceRequirements"    },    "restartCount": 0,    "runAsNonRoot": false,    "state": "running",    "stdin": true,    "stdinOnce": false, [...]
  33. 33. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – Using check_rancher2 - check_rancher2 is a monitoring plugin which uses Rancher 2’s API - Can run anywhere (requires http/https connection to API) - Checks status of: - Cluster(s) - Project(s) - Workload(s) (→ Services) - Pod(s) (→ Containers) - Consider the API endpoint as the “host”, check types as its services - Future (hopefully): - Workload or Pod usage (cpu, memory, network statistics) depends on https://github.com/rancher/rancher/issues/14230
  34. 34. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – check_rancher2 example - Workload stuck in “removing” - New workload (importer) already active - Monitoring needs to alert me about this!
  35. 35. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler AC – check_rancher2 example $ ./check_rancher2.sh ­H rancher2.example.com ­U token­ xxxxx ­P longsecretpass ­S ­t pod ­p c­r8ss7:p­85rmm ­o  importer­8bf85dcc9­r5rtn ­n gamma CHECK_RANCHER2 CRITICAL ­ Pod importer­8bf85dcc9­r5rtn is  removing|'pod_active'=0;;;; 'pod_error'=1;;;; - Plugin connects to Rancher 2 API using the information from the parameters: -H: API Host/DNS/IP -U: User-ID (token-xxxxx) -P: Password for User-ID -S: Use SSL (https) -t: Use “pod” check type -p: Project name (contains cluster ID, too) -o: Pod name (optional) -n: Namespace (optional, required for specific pod name)
  36. 36. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Application Containers – Recap - It’s not only Docker anymore → containerd (+ runc, kata) as container engine - An application container is not a classical host - Think of it as an application/process - Use orchestration/container management (Kubernetes, Rancher, OpenShift, ...) - Set up health checks → Healthchecks are your monitoring go go! - Monitor these health checks using orchestration/management API’s (Rancher2: check_rancher2) - There might also be plugins which use kubectl locally
  37. 37. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler References and links - Lost at sea: https://gcaptain.com/number-of-containers-lost-at-sea-falling-survey-shows/ - LXC: https://linuxcontainers.org/ - cgroup-v1: https://www.kernel.org/doc/Documentation/cgroup-v1/ - cgroup-v2: https://www.kernel.org/doc/Documentation/cgroup-v2.txt - Docker: https://www.docker.com/ - Kubernetes: https://kubernetes.io/ - containerd: https://containerd.io/ - Rancher: https://rancher.com/ - check_lxc: https://claudiokuenzler.com/monitoring-plugins/check_lxc.php - check_rancher2: https://claudiokuenzler.com/monitoring-plugins/check_rancher2.php
  38. 38. It’s all about the... containers! Monitoring containers OSMC 2018 Nuremberg @ClaudioKuenzler Thank you [[ $questions ­eq 0 ]] && exit 0

×