Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels and metadata about log streams. By using the same index and labels as Prometheus, Loki enables you to easily switch between metrics and logs, enhancing observability and streamlining the incident response process – a workflow we have built into the latest version of Grafana. In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days, but so far the response to the project has been overwhelming, with more the 6.5k GitHub stars and over 12hrs at the top spot on Hacker News. Loki is an opensource project, Apache licensed.

1. Grafana Loki: Like Prometheus, but for logs
Ganesh Vernekar, November 2019

2. About me
Presentation Title | 02
Ganesh Vernekar
Software Engineer, Grafana Labs
Prometheus Maintainer
Graduated this year from IIT Hyderabad
Twitter: @_codesome
Email: ganesh@grafana.com

4. 03/18 Project started
12/18 Launched at KubeCon NA
12/18 #1 on HN for ~12hrs!
03/19 Labels from logs
04/19 Lazy Queries; LogQL filtering
04/19 KubeCon EU: context, live tailing
06/19 v-.1.0 Beta release!
https://github.com/grafana/loki
Grafana Loki: Prometheus, but for logs | 04
goo.gl/5DEVH
6
Loki is horizontally-scalable, highly-available, multi-tenant log
aggregation system inspired by Prometheus

5. #0 Simple to scale
#1 Integrated with existing observability tools
#2 Airplane friendly and Cloud native
Grafana Loki: Prometheus, but for logs | 05

6. #0 Simple to scale
Grafana Loki: Prometheus, but for logs | 6

7. Grafana Loki: Prometheus, but for logs | 07
Existing log aggregation systems do full text indexing and support complex queries
DEwMGIwZ => {
time: “2018-01-31 15:41:04”,
job: “frontend”,
env: “dev”,
line: “POST /api/prom/push...”
}
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”

8. Grafana Loki: Prometheus, but for logs | 08
Existing log aggregation systems do full text indexing and support complex queries
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”
NodeN…Node1Node0

9. Grafana Loki: Prometheus, but for logs | 09
Loki doesn’t index the text of the logs, instead grouping entries into
“streams” and indexing those with labels.
{job=“frontend”, env=“dev”} => {
time: “2018-01-31 15:41:04”,
line: “POST /api/prom/push HTTP/1.1 502 0"
}

10. #1 Integrated with existing
observability tools
Grafana Loki: Prometheus, but for logs | 10

11. Grafana Loki: Prometheus, but for logs | 11
1. Alert 2. Dashboard 3. Adhoc Query
4. Log Aggregation5. Distributed TracingFix!

12. Prometheus’ data model is very simple:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Identifiers are bags of (label, value) pairs:
{job=“foo”, instance=“bar”, ... }
Timestamps are millisecond int64, values are float64
Grafana Loki: Prometheus, but for logs | 12
https://www.slideshare.net/Docker/monitoring-the-prometheus-way-julius-voltz-prometheus

13. Grafana Loki: Prometheus, but for logs | 13
AppsAppsAppsapps
k8s
#0 Prometheus talks to k8s to discover list of targets
#1 Target information is “relabelled” to build labels
#2 Metrics are pulled from apps
#3 Target labels added to series labels

14. Loki’s data model is very similar:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Timestamps are nanosecond floats, values are byte arrays.
Identifiers are the same - label sets.
Grafana Loki: Prometheus, but for logs | 14

15. Grafana Loki: Prometheus, but for logs | 15
prom
tail
AppsAppsAppsapps
k8s

16. Grafana Loki: Prometheus, but for logs | 16

17. Grafana Loki: Prometheus, but for logs | 17
1. Alert 2. Dashboard 3. Adhoc Query
4. Log Aggregation5. Distributed TracingFix!

18. #2 Airplane friendly and Cloud
Native
Grafana Loki: Prometheus, but for logs | 18

19. Grafana Loki: Prometheus, but for logs | 19
Airplane friendly
prom
tail
Apps
Apps
Apps

20. Grafana Loki: Prometheus, but for logs | 20
Scale out
prom
tail
Apps
Apps
Apps

21. Grafana Loki: Prometheus, but for logs | 21
Scale out
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps

22. Grafana Loki: Prometheus, but for logs | 22
Microservices
promtail

23. #0 Simple to scale
#1 Integrated with existing observability tools
#2 Cloud native and airplane friendly
Grafana Loki: Prometheus, but for logs | 23

24. Bonus

25. Loki as a Prometheus datasource!
(From v0.4.0)
Grafana Loki: Prometheus, but for logs | 25

26. DEMO

27. Thanks! Questions?