When moving your apps to Kubernetes, you need to keep your existing observability at the same level or better. Kubernetes will give you some challenge, as you can’t strictly deploy the TICK Stack as you did before, but also allow some opportunities. The talk is about my journey on this topic and will cover Telegraf as DaemonSet to fetch nodes resources, as a deployment to fetch metrics from different endpoints and hopefully with Telegraf as an operator to illustrate sidecar deployment. All these metrics will be pushed to InfluxDB (v1/v2) and may be visualized in Chronograf or Grafana.

1. Nicolas Steinmetz - CérénIT
Sustainable
Observability
From Bare Metal
To k8s world

2. © 2020 InfluxData. All rights reserved. 2
$ whoami?
∙Freelance at CérénIT: Architecture, Automation & Time Series
∙TIG & TICK user for 5 years (InfluxDB 0.9.x)
∙InfluxAce for France since 2019
∙Host of the Paris Time Series Meetup (ptsm.io)
∙Co-Host of the BigDataHebdo(.com) podcast

3. And why the TICK/TIG stack ?
Some principles to start with...

4. How did I get
there ? ∙Custom metrics for home sensors &
extended to platform monitoring
∙Best of breed platform
∙Nice UI and Dashboards
∙Ready to use
∙Python API (pre-telegraf world 😉)
∙Raspberry Pi compatible

5. ∙Monitoring outside the platform
∙Telegraf to collect and send metrics
to InfluxDB
∙Push mechanism
∙Grafana for Alerting and Visualising
My monitoring
principles

6. It’s a journey…

7. Once upon a time...
From bare metal to containers...

8. ∙(Ab)use of /etc/telegraf/telegraf.d
∙Automated by Infrastructure as
Code
∙Host and application metrics 🤩

9. ∙Docker inputs plugin provides only
general metrics (mem, cpu, net,
healthcheck)
∙Lost visibility on what happens
inside the container 😰
Hello Docker !

10. ∙Add telegraf in docker ! 😌
∙Get metrics back again for services
∙But not the perfect solution…
∙A new pattern is rising… 😏

11. To infinity and beyond...
From containers to kubernetes...

12. Monitoring in
k8s world ?
∙Nodes: master(s) & workers
∙Kubernetes Core Services (etcd,
coredns, etc)
∙Application and related kubernetes
resources

13. ∙De facto standard
∙Ecosystem relies on prometheus
(exporters)
∙Prometheus Operator
∙Node metrics
∙Core service metrics
∙Application metrics
∙Grafana dashboards
∙Alert Manager & Prom UI
Prometheus

14. ∙Already have TIG !
∙Not another / custom kubernetes
tool
∙Long term storage ?
∙Analytics ?
∙Not embedded in the cluster nor
want to enable pull monitoring from
outside
So why not just
choose
prometheus ?
🤨🤔

15. Can we get the same with
TIG/TICK Stack ?

16. ∙Do we want to have exactly the
same data or something similar ?
∙Just not try to duplicate Prom
Operator dashboards but question
your needs and the existing
dashboards.
∙Available metrics may depend of
your kubernetes provider
Before diving !

17. ∙Contributed an updated version of
telegraf-ds helm chart 💪
∙Mix of traditional plugins +
kubernetes input plugin
∙Opinionated default configuration
Global & Node
metrics ?

19. ∙Inspired from Prometheus Operator
Dashboards
∙Reproduced the one that interested
me and extended them
∙Most of the metrics are identical ; a
few are different

20. ∙Telegraf helm chart
∙kube_inventory input plugin
∙Prometheus input plugin
∙All inputs plugin for your services
∙Telegraf.conf = ConfigMap
Application
metrics ?

21. ∙Telegraf-operator (alpha) chart
∙Inject a telegraf container as sidecar
pod
∙Telegraf classes to define
configuration to apply
∙Interesting for non service metrics
∙Take care of Telegraf proliferation
Telegraf
operator

22. ∙Young and promising initiative
∙Ready to use & Dashboard as code
∙Kubernetes dashboards by
bonitoo.io
∙Based on kubernetes and
kube_inventory input plugins
∙github.com > influxdata >
community-templates > k8s
InfluxDB
Community
templates

24. ∙Take only node-exporter,
kube-state-metrics from
Prometheus Operator
∙Use telegraf to collect prometheus
metrics via prometheus input plugin
A third way ?

25. Next steps and take away

26. ∙Explore deeper InfluxDB 2.0,
especially to dissociate Alerting from
Dashboards
∙Explore deeper telegraf-operator for
in pod metrics to confirm my
hypothesis
Next steps

27. ∙Possible to monitor kubernetes
platform with telegraf
∙Don’t need to deploy prometheus in
your cluster
∙Leverage prometheus exporters
with prometheus input plugin
∙Mix of Prometheus Operator and
Telegraf to have best of both worlds
?
∙Watch progress of telegraf operator
Take away

28. Thanks for watching !

29. © 2020 InfluxData. All rights reserved. 29
Nicolas Steinmetz - CérénIT
Personal : @nsteinmetz
Company : @_CerenIT ; https://www.cerenit.fr/ ; contact@cerenit.fr
Paris Time Series Meetup (mostly in French) : https://www.ptsm.io/
Podcast (in French) : https://bigdatahebdo.com/
@nsteinmetz on InfluxData Slack

30. © 2020 InfluxData. All rights reserved. 30