My Ignite for the Open Source Monitoring Conference 2021 in Nuremberg

1. OSMC
Kris Buytaert
@krisbuytaert
Pipeline all the Dashboards as Code ?
November , 2021

2. • I used to be a developer
• Then I became an Ops person
• Chief Trolling/Travel/Technical Officer @ Inuits.eu
• Everything is a Freaking DNS Problem
• DNS : devops needs sushi
• Buiding Clouds since before the Bookstore
• Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ...
Who am I ?
@krisbuytaert

3. • Operational Metrics
• Application Metrics
• Business Metrics
• Travel time to home during Traffic Jams
• Time between last conference
• Legobricks ordered
• Sandwiches ordered by the team
Dashboard all the things
@krisbuytaert

4. • People edit a Dashboard
• People clone a Dashboard
• People create a new Dashboard
• People "improve" other people's Dashboards
• People delete other people's Dashboards
class {'chaos':
ensure => present
}
Dashboards Sprawl
@krisbuytaert

5. • No more hacking in production
No more manual changes?
@krisbuytaert

6. What do we want
@krisbuytaert

7. Jsonnet is a DSL to build JSON Data,
JSON On steroids
Jsonnet is dead
Jsonnet is alive
Grafonet
GrafanaLib
Dashboard as Code
@krisbuytaert

8. Look a Squirrel !
@krisbuytaert

9. We are all JSON engineers
We are all YAML engineers
The average end user who wants to create his own dashboard
is not a devoops YAML Engineer
This is just too complex
Problems with Dashboard as Code
@krisbuytaert

10. • A dev Grafana
• Dashboard as Code
• Dashboard for Dirty Clickers
• A prod Grafana (read only)
• A way to promote from dev to prod
What we really need
@krisbuytaert

11. https://github.com/inuits/dashboard-manager
A command line utility to manage and promote dashboards between Grafana
instances, coupled with gitlab-ci.
Dashboard Manager for Grafana
@krisbuytaert

12. grafana_instances_output:
- api_key_file: production-secret
url: http://127.0.0.1:3000
name: prod
grafana_instances_input:
- api_key_file: dev-secret
url: https://remote-dev.example.com/
name: dev
http_client:
# Configured like alertmanager http client:
# https://prometheus.io/docs/alerting/latest/configuration/#http_conf
tls_config:
insecure_skip_verify: true
Example Configuration
@krisbuytaert

13. fetch --output-directory=OUTPUT-DIRECTORY
Fetch dashboards from input grafana.
compare --dashboards-directory=DASHBOARDS-DIRECTORY --results=RESULTS
Compare dashboards.
upload --dashboards-directory=DASHBOARDS-DIRECTORY --input-instance=INPUT-I
--dashboards=DASHBOARDS
Upload dashboards.
Usage
@krisbuytaert

14. Step 1 : User creates Dashboard in dev
@krisbuytaert

15. Step 2 : Trigger the pipeline
@krisbuytaert

16. Step 3 : Check downstream Created Jobs
@krisbuytaert

17. Step 4 : Promote the (individual) Dashboard
@krisbuytaert

18. Step 5 : New dashboard is in prod
@krisbuytaert

19. • Usable for End Users
• Version controlled
• Reproducable
• Preventing dashboard Sprawl
• Can still be used for dashboards from "real" code. to do the promotions
Conclusion
@krisbuytaert

20. Kris Buytaert
@krisbuytaert
kris@inuits.eu
Essensteenweg 31
2930 Brasschaat
Belgium
Contact:
info@inuits.eu
+32-3-8082105