Discover How Volvo Cars Uses a Time Series Database to Become Data-Driven

InfluxDB and TICK Stack @ Volvo Cars
InfluxDB and TICK Stack @ Volvo Cars. Maksim Puzyrkov, 2020

MAKSIM PUZYRKOV
DevOps Engineer at Volvo Cars
Maksim has almost 20 years of experience in the IT field. During his career, he
worked with a broad range of technologies. Now his professional interests lie
within Enterprise Management and Monitoring solutions.
Maksim works as an Engineer at Volvo with the DevOps Enablement product;
the team adopts an agile way of working. He was responsible for the
development of Volvo’s monitoring approach over the years and contributed to
a number of transformations and platform changes based on growing
company needs.
Recently, Maksim has been working with integration of InfluxDB and other
solutions in the monitoring platform.

IBM Tivoli Framework &
Tivoli Enterprise Console
IBM Tivoli Monitoring &
Tivoli Netcool Omnibus
TICK Stack + Elastic Stack +
Grafana +
IBM Netcool Operations Insight
History of Monitoring At Volvo Cars: Monitoring tools evolution

Challenges of Monitoring
Low Transparency
Non-Collaborative Mindset
Lack of Self-Service

Agile Transformation & DevOps
[ ] Transparency
[ ] Mindset
[ ] Self-Service

Empowered
Autonomous
Digital Teams
We Provide Tools that support Methodology
to accelerate DevOps culture for digital product teams

What is DevOps Enablement at Volvo Cars?
It is about People, Processes and Tools. We offer technologies, ways of
working and thinking that support teams in their DevOps journey.
How do we help to get started?
We help to set up and support in how to structure, plan and operate in a
DevOps mode. As well we provide tool chains that include all functionalities
in order to perform as a DevOps organization.
DevOps Enablement at Volvo Cars
What do we provide?
A toolchain with all the elements needed to develop great digital
products. From planning to release to operations.
What is the transformation we support?
At Volvo Cars we are moving to lean and agile teams. These teams
will work self-sufficient. The DevOps Enablement team provides all
necessary parts in order to perform this change.

DevOps Enablement Principles
Best of breed
VS
One fits all
Enable Digital Teams
=
Provide Self-Service
Accelerate by
removing dependency
Don’t rule
Provide more value than
available on the internet

Logging
Elasticsearch Logstash Kibana
Monitoring &
Analytics IBM NOI Grafana Application Insights Apica
Quality
Performance Center Load Runner UFT Azure Test Plans SonarQube
CI/CD
Azure Repos Azure Pipelines Azure Artifacts
Planning
Azure Plans
Environment
Azure Cloud / On-Premise Containers Ingress and API Mgmt. Automation ServiceNow
DevOps Enablement Stack

OR “Classic” VS “Modern”
“TICK Stack is a loosely coupled yet tightly integrated set of open source projects designed to handle massive
amounts of time-stamped information to support your metrics analysis needs.”
Why InfluxDB and TICK Stack?

The TICK Stack is a set of Open-
Source projects designed to handle
massive amounts of time-stamped
information for metrics analysis
needs.
• Telegraf – agent for collecting
and reporting metrics.
• InfluxDB – datastore optimized
for time-stamped data.
• Chronograf – administrative and
visualization user interface.
• Kapacitor – real-time data
processing and alerting engine.
What is TICK Stack?

• Grafana – multi-source
visualization interface.
• 3rd party integrations – push/pull
telemetry data from other
systems at DevOps Enablement.
• Custom agents and scripts –
purpose-built agents and scripts
to gather data from cloud
providers, SaaS and in-house
tools.
TICK Stack satellites at DevOps Enablement

“Classic” Monitoring Solutions VS InfluxDB or time-series
SELECT used FROM mem WHERE “env”=‘prod’ GROUP BY “app”, “location”
measurement=mem,field=used,app=“app A”,location=“emea”,env=“prod” 23,25,22…
measurement=mem,field=used,app=“app B”,location=“emea”,env=“prod” 45,64,80…
measurement=mem,field=used,app=“app C”,location=“americas”,env=“prod” 36,65,49…
10:10 10:20 10:30 10:40 10:50
app A
emea
americas app C
prod
dev app A
app B
prod
memory usage, %

Data Gathering

Data Collectors & Connectivity
API Management (3scale)
‒ Auth keys generated through self-service portal
‒ Unique keys helps users maintain application
groups
Kafka Gateways
‒ Very easy, only network connection required
‒ Adjustable buffer that can keep data for days
‒ Supported by Telegraf outputs
Agent of User Choice
‒ Telegraf, Fluentd, Prometheus exporter, custom
script or App built-in metric reporter
‒ Protocols and data formats:
InfluxDB Line Protocol | JSON | CSV | Collectd | Dropwizard | Graphite
| Logfmt | Nagios | OpenTSDB | Prometheus | Value | Wavefront
Data Mover
InfluxDB
Kafka Cluster
Host
Cloud App
Firewall
Containers
API Management

Telegraf Installation and Configuration
• Ansible role for Telegraf deployment
• Jump-start configuration packages
• Recommended configuration files structure
C:Program Filestelegraf
| telegraf.conf
| telegraf.exe
|
---telegraf.d
aggregators.conf
inputs_system.conf
inputs_application_a.conf
inputs_application_b.conf
outputs_kafka.conf
processors.conf

Alerting

Event Management and Alerting Self-Service
Alert Management API via 3scale
‒ Auth key through self-service portal
‒ Extended logging and stats per application
Alerts from many sources
‒ Correlation and de-duplication
‒ Situation awareness at a glance
Auto ticketing via TICK script config or central config
‒ Alerts can be extended with ticketing attributes
‒ Extra rules defined on a central event router
Elasticsearch as historical database
‒ Trend analysis for alerts and incidents
Not restricted to “standard” alert management process
‒ Users can use bundled alert handlers
TICK Scripts Blueprints and Templates InfluxDB
3scale
API Management
Kapacitor
IBM NOI
ServiceNow
Elasticsearch

Keys For Efficient Alerting: Tagging Strategy & Self-Services
InfluxDB
Group “EMEA” in Critical status!
IBM NOI
Application A, EMEA
Application A, APAC
Application B, AMERICAS
Application C, EMEA
Group by “Region”
Kapacitor
• Tagging strategy
• Soft enforcement through review
and recommendations
• Learning and knowledge sharing
• Templates and blueprints

|from()
.database('telegraf')
.retentionPolicy('autogen')
.measurement('disk')
.where(lambda: "environment" == ’prod’ and "path" =~ /data.*/ )
.groupBy('host', 'path')
|mean('used-percent')
.as('stat’)
|alert()
.id('{{ index .Tags "host"}}/disk-used/{{ index .Tags "path" }}')
.message('{{ .ID }}:{{ index .Fields "stat" }}')
.warn(lambda: "stat" > 75)
.crit(lambda: "stat" > 90)
.details(
"TicketFlag": "true",
"TicketPriority": "3",
"TicketGroup": "DevOps Enablement")
This example automatically detects
new metrics received from
production servers.
Warning and Critical alerts triggered
if average disk usage crossed 75 or
90% threshold accordingly.
Separate alerts generated for unique
combination for path and host name.
Alert rule example in TICK Script language

Architecture and Automation

TICK Stack at Volvo Cars
Telegraf & other
data collectors
3scale
API Management
Kafka Clusters
Application
Gateway
Ingest Telegrafs Short Term DB
RAW Data
Long Term DB
Pre-filtered Data
Ingest Kapacitors
Replicate measurements
Alerting Kapacitors
HA Cluster
Private Kapacitors
Dev env for users
TICKscript
Replicator
PROD
DEV
Watcher
InfluxDB
& Kapacitor
Data Mover

TICK Stack at Volvo Cars: filter data
Ingest Telegrafs
Short Term DB
RAW Data
Long Term DB
Pre-filtered Data
Ingest Kapacitors
Replicate measurements
Alerting Kapacitors
HA Cluster
Private Kapacitors
Dev env for users
TICKscript
Replicator
PROD
DEV
Azure Pipeline Azure Repo
Watcher
InfluxDB
& Kapacitor
Permission List

TICK stack at Volvo Cars: Kapacitors – Blue/Green deployment
Azure Kubernetes Service
Application Gateway
service service service
Traefik Ingress Controller
serviceserviceservice
Azure
File share

TICK stack at Volvo Cars: Kapacitors – Infrastructure as Code
Azure Kubernetes Service
Application Gateway
service service service
serviceserviceservice
Terraform
Azure
File share

“Classic” VS “Modern” Monitoring Life-Cycle

Monitoring Life-Cycle: New Monitoring Rule or Graph
Create new Monitoring Template
Assign Monitoring template
to Group of Objects (servers or apps)
Alerts Generated
by New Template Rules
Dashboards with Data
from new Template
“Classic”
Create new Monitoring Rule
Alerts Generated
by Kapacitor service
Dashboards with Metrics
from Agent configuration
No need to apply or reconfigure
agent to gather additional metrics.
Most of the metrics
collected by Telegraf by default.
Agent Discover and Apply
Monitoring Template
“Modern”

Agile Transformation & DevOps
[V] Transparency
[V] Mindset
[V] Self-Service

Discover How Volvo Cars Uses a Time Series Database to Become Data-Driven

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Discover How Volvo Cars Uses a Time Series Database to Become Data-Driven

Similar to Discover How Volvo Cars Uses a Time Series Database to Become Data-Driven (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

Discover How Volvo Cars Uses a Time Series Database to Become Data-Driven