1.
Observability with
InfluxDB/IOx &
OpenTelemetry
Jacob Marble, InfluxData

2.
© 2021 InfluxData Inc. All Rights Reserved.
About Me

3.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
InfluxDB/TSM: Definition
● Columnar Series-oriented storage engine
○ Like a log-structured merge-tree
■ Key is roughly concat(tags, field key, timestamp)
■ Value is field value
○ “Column per series”
○ Leverages mmap() syscall
● Series index
○ Index to data by concat(tags, field key)
● Introduced in 2015
○ Superseded engine based on BoltDB
● Implemented in Golang

4.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
InfluxDB/TSM: Good
The TSM storage engine is good at:
● Storage capacity: hundreds of GBs
● Storage cardinality: millions of series
● Query limits: selective, analytic
● Strong use cases: IoT, metrics, events*

5.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
InfluxDB/IOx: Definition
● Distributed columnar storage engine
○ Parquet
■ Columns are typed data columns
■ Time is a column type
■ Tags, fields: semantic property of columns
○ Arrow, Flight
■ In-memory, network
○ Federated storage, write, query
■ Collection, storage, query
● No series index
○ Partition keys
○ Dictionary encoding
● Under development since 2020
● Implemented in Rust

6.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
InfluxDB/IOx: Better
The IOx storage engine is good at:
● Storage capacity: unlimited
● Storage cardinality: unlimited
● Query limits: network protocols
● Strong use cases: IoT, metrics, events, logs, traces, ...

7.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: Definition
All observability signals in one standard.
✅ Tracing
✅ Logs
✅ Metrics
The project has wide industry support.
● Top-tier cloud vendors and APM SaaS are committed.
● SDKs for ~10 languages.
● Second most active CNCF project in 2020.
○ Most active was Kubernetes.

8.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
Open Source Observability: Today
metrics logs traces
instrumentation Prometheus printf() Jaeger
collection Prometheus Expo Logfmt, JSON Thrift or gRPC
delivery Pull via HTTP Push via stdout Push via UDP
pipeline Telegraf Fluentd Jaeger
storage InfluxDB MongoDB Elasticsearch
query Grafana Graylog Jaeger

9.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
Open Source Observability: Today
application
application
application
application
application
application
application
application
Telegraf
Fluentd
Jaeger
InfluxDB
MongoDB
Elasticsearch
Grafana
Graylog
Jaeger
SDK
SDK
Prom.
Expo
Logfmt
Jaeger
Thrift
SDK

10.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: Tomorrow
metrics logs traces
instrumentation OpenTelemetry SDK
collection OTLP
delivery Push via gRPC
pipeline OpenTelemetry Collector
storage
?
query

11.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: Tomorrow
application
application
application
application
application
application
application
application
OTel SDK
?
OTLP
OTel
Collector

12.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry SDK: Versatile
application
application
application
application
application
application
application
application
OTel SDK
Telegraf
Fluentd
Jaeger
Prom.
Expo
Logfmt
Jaeger
Thrift
InfluxDB
MongoDB
Elasticsearch
Grafana
Graylog
Jaeger

13.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Collector: Versatile
OTel
Collector
InfluxDB
MongoDB
Elasticsearch
Grafana
Graylog
Jaeger
application
application
application
application
application
application
application
application SDK
SDK
SDK
Prom.
Expo
Logfmt
Jaeger
Thrift

14.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Migration: Incremental
InfluxDB
MongoDB
Elasticsearch
Grafana
Graylog
Jaeger
application
application
application
application
application
application
application
application
OTel SDK
OTLP
OTel
Collector

15.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry SDK: Span ID Everywhere
application
application
application
application
application
application
application
application
span := newSpan(parent)
defer span.Finish()
log.Warn("error...")
OTel SDK
span
ID: 3849
log record
spanID: 3849
OTel
Collector

16.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Collector: Temporal Aggregation
OpenTelemetry
Collector
-
bar = avg(15m)
metric: foo
gauge 10s
metric: bar
gauge 15m

17.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Collector: Spatial Aggregation
metric: foo
labels: k8s_node
metric: bar
labels: ∅
OpenTelemetry
Collector
-
bar = sum()

18.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Collector: Traces→Metrics
span: foo
duration: 456ms
metric: bar
histogram
OpenTelemetry
Collector
-
if (span = foo) then
bar = hist(duration)

19.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry Collector: Tail Sampling
span: foo
duration: 456ms
trace
sampled
OpenTelemetry
Collector
-
if (span = foo &&
duration > 100ms)
then
sample trace

20.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: Vendor Neutral
application
application
application
application
application
application
application
application
OTel SDK
?
OTel
Collector
OTLP

21.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: SaaS APM
application
application
application
application
application
application
application
application
OTel
Collector
OTel SDK
$aa$
Very Fancy UI
OTLP
OTLP

22.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
OpenTelemetry: Open Source End-to-End
application
application
application
application
application
application
application
application
OTel SDK
Something
People Want
Object Storage
InfluxDB/IOx
OTel
Collector
OTLP
OTLP

23.
© 2021 InfluxData Inc. All Rights Reserved.
© 2021 InfluxData Inc. All Rights Reserved.
InfluxData 💜 OpenTelemetry: Status
● InfluxData common schema for observability
○ github.com/influxdata/influxdb-observability
● OTel Collector sends Line Protocol
● OTel Collector receives Line Protocol
● Telegraf sends OTLP
● Telegraf receives OTLP
● Jaeger UI queries IOx (prototype)

24.
© 2021 InfluxData Inc. All Rights Reserved.
Questions

25.
© 2021 InfluxData Inc. All Rights Reserved.
Thank you
Jacob Marble, InfluxData
jacobmarble@influxdata.com