Alluxio Day IV June 24, 2021 https://www.alluxio.io/alluxio-day/ Speaker: Pan Liu, Tencent

1. Setting up Monitoring System for Alluxio
with Prometheus and Grafana in 10 minutes
Pan Liu @ Tencent
2021.06.24

2. • Pan Liu
• Tencent
• Presto on Alluxio

3. Contents
• Part 1. How Alluxio Metrics System
Works
• Part 2. How to Implement a custom
Alluxio Sink
• Part 3. How to Set Up Monitoring of
Alluxio in 10 mins

4. Part 1. How Alluxio Metrics System Works

5. Alluxio Metrics System
• Framework
• Types of metrics
• Name metrics
• Flow of metrics from worker to master

6. Alluxio Metrics System
• Framework
Source1
Metrics
system
Sink1
Source2 Sink2
… …

7. Alluxio Metrics System
• Two Types of Metrics:
• Cluster Metrics : Aggegated from Workers and Clients
• Process Metrics: Collected by Each Alluxio Process
Master
Cluster Metrics
Process Metrics
Worker
Worker
Worker
Client
Client

8. Alluxio Metrics System
• Metrics Name:
• Master: Master.[metricName].[tag1].[tag2]...
• Non-Master: [processType].[metricName].[tag1].[tag2]...[hostName]
Master.GetFileInfoOps
Worker.OpenExistingFile.User:user.UFS:hdfs:%2F%2F9%2E135%2E88%2E92:9000%2F.UFS_TYPE:hdfs.worker_host

9. Alluxio Metrics System
MetricMaster
BlockMaster
processWorkerMetric
workerHeartbeat
MetricsStore putWorkerMetrics
RpcServer
BlockWorker
BlockMasterSync
BlockMasterClient
heartbeat
RpcClient
MasterProcess WorkerProcess
heartbeat
blockHeartbeat
blockHeartbeat

10. Part 2. How to Implement a custom Sink of Alluxio

11. Alluxio Sinks
• Passive VS. Active
• Passive: service PULL
• Active: report periodically PUSH
Active Passive

12. Alluxio Sinks
Active: Report periodically
Master_host:port/metrics/json
./logs/master.out
Passive: Scratch from server

13. Active Sink: ConsoleSink
…
…

14. Passive Sink : PrometheusMetricsServlet
• getHandler() called by master or worker
• Work as service
• Metrics are available only when requested
E.g. MasterProcess

15. Sink Extension
• Passive or Active ?
• XmlSink as an example of Active type: Print metrics in XML format to a specified path.
• Implement Sink interface
• Construct a XmlReporter to report metrics in XML format
• Config to enable XmlSink
conf/metrics.properties

16. Part 3. How to Set Up Monitoring for alluxio in 10
minutes

17. Alluxio Web UI Monitoring
Web UI metrics page
Convenient but doesn’t work well sometimes…

18. Alluxio Monitoring with Prometheus and Grafana
How it works ?
1. Prometheus scrapes metrics from Alluxio servers and transforms to time series data
2. Grafana server get metrics using the PromQL
3. Grafana web UI displays metrics in dashboards
What we need to do ?
1. Install and start Prometheus and Grafana Server
2. Add Alluxio Jobs to Prometheus
3. Download and import Grafana dashboard
4. Modify variables of dashboard template

19. Alluxio Monitoring with Prometheus and Grafana
prometheus.yml Version:
- Prometheus Version: 2.22.2
- Grafana Version: 7.5.6
- Alluxio Version: 2.5.0-3
targets/local/masters/master_01.yml
targets/local/workers/worker_01.yml
Services:
- Prometheus Server @ master:9090
- Grafana Server @ worker:3000
- Alluxio Master @ master
- Alluxio Worker @ worker

21. Alluxio Monitoring with Prometheus and Grafana

22. Alluxio Monitoring with Prometheus and Grafana
• Alluxio IO Key Metrics
• Read Local & Read Remote & Alive workers …
• Storage
• Space Used & UFS Space Used …
• Workers Blocks
• Cached Blocks & Evicted Blocks …
• Logical Operations
• Mount Operations & File Pinned …
• Alluxio Metadata Operations
• Block Hearbeat Cost & Get status Cost …
• AsyncCache Blocks & Operations
• AsyncCacheSuccessedBlocks & AysncCacheFailedBlocks …
• Master JVM Memory
• Master Heap Memory & Total Memory …

23. Alluxio Monitoring with Prometheus and Grafana
• Alluxio docs:
https://docs.alluxio.io/os/user/edge/en/operation/Metrics-
System.html#grafana-web-ui-with-prometheus
• Grafana dashboards:
https://grafana.com/grafana/dashboards/13467

24. Thanks!
panyliu@tencent.com