SlideShare una empresa de Scribd logo
1 de 18
1Benchmarking and comparison of kernel and userspace tracers
Benchmarking and comparison of
kernel and userspace tracers
5 May, 2016
École Polytechnique de Montreal
Mohamad Gebai
Michel Dagenais
2Benchmarking and comparison of kernel and userspace tracers
Content
The tracers
The setup
The benchmarks
Results
Kernel space
System calls
Filtering
Userspace
3Benchmarking and comparison of kernel and userspace tracers
The tracers
Kernel tracers
LTTng
Ftrace
Perf
eBPF (kind of)
Userspace tracers
LTTng
Printf()
LTTng using tracef()
Extrae
Lightweight homemade basic tracer
4Benchmarking and comparison of kernel and userspace tracers
The setup
Intel i7 @ 3.40 GHz
16 GB of RAM
Linux kernel version 4.5.0
LTTng suite built from tip of the branch stabe­2.7
Deactivated:
Hyperthreading
C­states
CPU idle
Intel Turbo Boost
LTTng
Snapshot mode
Subbuffer size 32 KB
Ftrace
Global clock
5Benchmarking and comparison of kernel and userspace tracers
The benchmarks
Objective: measure the cost of writing a tracepoint
Kernel tracers:
Microbenchmark in kernel space
Payload: 32 bytes
Userspace tracers:
Microbenchmark in userspace
Payload: 32 bytes
Steady state (drop initial page faults, cache misses, initialization, etc.)
Do a full buffer write and start overwriting
6Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: the outline
Final numbers:
Average 90th
percentile
Value (ns) Overhead Value (ns) Overhead
None 17 0% 17 0%
LTTng 92 441% 89 424%
Ftrace 118 594% 114 571%
Perf 120 606% 118 594%
LTTng-kprobe 122 618% 120 606%
eBPF 131 671% 126 641%
None LTTng Ftrace Perf LTTng-kprobe eBPF
0
20
40
60
80
100
120
140
Average latency of a tracepoint
Latencyinns
7Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: the outline
Final numbers:
None LTTng Ftrace Perf
0
50
100
150
200
250
300
Average of tracepoint latency in steady vs transient states
Transient
Steady
Kernel tracers
Latencyinns
8Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: LTTng vs Ftrace
9Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: closer look
10Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: closer look
11Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: ioctl() syscall
Final numbers:
Average 90th
percentile
Value (ns) Overhead Value (ns) Overhead
None 99 0% 98 0%
Ftrace 311 214% 302 208%
LTTng 341 244% 333 240%
Perf 600 506% 586 498%
None Ftrace LTTng Perf
0
100
200
300
400
500
600
700
Average latency of a syscall with tracing
Kernel tracers
Latencyinns
12Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: open() syscall
0 10 20 30 40 50 60
0
100
200
300
400
500
600
700
800
900
1000
Latency of tracing syscall open() according to filename length
LTTng
Ftrace
Filename length in bytes
Latencyinns
13Benchmarking and comparison of kernel and userspace tracers
Kernel tracing: filtering with LTTng
0 10 20 30 40 50 60
0
50
100
150
200
250
300
350
400
450
Average latency of a tracepoint with filtering
failed conditions
average latency on success
no filter
Filter bytecode size (number of conditions)
Latencyinns
14Benchmarking and comparison of kernel and userspace tracers
Userspace tracing: the outline
Final numbers, but...
LTTng is signal­safe
reentrant
LTTng is scalable
Average 90th
percentile
Value (ns) Overhead Value (ns) Overhead
None 18 0% 17 0%
LW-ust 67 272% 64 276%
Extrae 159 783% 118 594%
Printf 245 1261% 243 1329%
LTTng 255 1317% 249 1365%
LTTng tracef() 536 2878% 523 2976%
None LW-ust Extrae Printf LTTng LTTng tracef()
0
100
200
300
400
500
600
Average latency of a tracepoint
Latencyinns
15Benchmarking and comparison of kernel and userspace tracers
Userspace tracing: closer look
16Benchmarking and comparison of kernel and userspace tracers
Userspace tracing: scalability
17Benchmarking and comparison of kernel and userspace tracers
Related work
More scalability runs
Closer look at aggregators (eBPF, Systemtap)
18Benchmarking and comparison of kernel and userspace tracers
Acknowledgements
Professor Michel Dagenais
Julien Desfossez
Mathieu Desnoyers
Ericsson

Más contenido relacionado

Destacado

LTTng (EclipseCon 2010)
LTTng (EclipseCon 2010)LTTng (EclipseCon 2010)
LTTng (EclipseCon 2010)Fchouinard
 
Bridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracingBridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracingChristian Babeux
 
Advanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngAdvanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngmarckhouzam
 
Ancillary Research Task - Analysis of Advertisements and Digipaks
Ancillary Research Task - Analysis of Advertisements and DigipaksAncillary Research Task - Analysis of Advertisements and Digipaks
Ancillary Research Task - Analysis of Advertisements and DigipaksJerard Ocitti
 
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシIoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ新潟コンサルタント横田秀珠
 
Site jornal apresentação
Site jornal apresentaçãoSite jornal apresentação
Site jornal apresentaçãoAlden Aguiar
 
Venid a mí todos los que estáis trabajados y cargados
Venid a mí todos los que estáis trabajados y cargadosVenid a mí todos los que estáis trabajados y cargados
Venid a mí todos los que estáis trabajados y cargadosmisionero2000
 
экскурсии в алапаевский район (боярская а.)
экскурсии в алапаевский район (боярская а.)экскурсии в алапаевский район (боярская а.)
экскурсии в алапаевский район (боярская а.)URFU
 
Отдельные улицы и кварталы
Отдельные улицы и кварталыОтдельные улицы и кварталы
Отдельные улицы и кварталыURFU
 
Educ.ar tutorial slide share
Educ.ar   tutorial slide shareEduc.ar   tutorial slide share
Educ.ar tutorial slide sharenatis1906
 
Clima e tempo cap 14 15 16
Clima e tempo cap 14 15 16Clima e tempo cap 14 15 16
Clima e tempo cap 14 15 16Christie Freitas
 
Você é preconceituoso? Descubra com essa série fotográfica de Joel Pares
Você é preconceituoso? Descubra com essa série fotográfica de Joel ParesVocê é preconceituoso? Descubra com essa série fotográfica de Joel Pares
Você é preconceituoso? Descubra com essa série fotográfica de Joel ParesEmerson Jonas
 
De teamrollen van Belbin in makkelijk te onthouden steekwoorden
De teamrollen van Belbin in makkelijk te onthouden steekwoordenDe teamrollen van Belbin in makkelijk te onthouden steekwoorden
De teamrollen van Belbin in makkelijk te onthouden steekwoordenIsabelle Langeveld
 
Management by Objectives
Management by ObjectivesManagement by Objectives
Management by ObjectivesAgnes Miriam
 
LTTng-UST: Efficient System-Wide User-Space Tracing
LTTng-UST: Efficient System-Wide User-Space TracingLTTng-UST: Efficient System-Wide User-Space Tracing
LTTng-UST: Efficient System-Wide User-Space TracingChristian Babeux
 
Tema 9 la segunda república española (1931 1936) (1)
Tema 9 la segunda república española (1931 1936) (1)Tema 9 la segunda república española (1931 1936) (1)
Tema 9 la segunda república española (1931 1936) (1)Mercedes Espinosa Contreras
 

Destacado (20)

LTTng (EclipseCon 2010)
LTTng (EclipseCon 2010)LTTng (EclipseCon 2010)
LTTng (EclipseCon 2010)
 
Bridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracingBridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracing
 
Advanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngAdvanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTng
 
Ancillary Research Task - Analysis of Advertisements and Digipaks
Ancillary Research Task - Analysis of Advertisements and DigipaksAncillary Research Task - Analysis of Advertisements and Digipaks
Ancillary Research Task - Analysis of Advertisements and Digipaks
 
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシIoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ
IoT(モノのインターネット)活用セミナー(群馬県)高崎商工会議所チラシ
 
Site jornal apresentação
Site jornal apresentaçãoSite jornal apresentação
Site jornal apresentação
 
Venid a mí todos los que estáis trabajados y cargados
Venid a mí todos los que estáis trabajados y cargadosVenid a mí todos los que estáis trabajados y cargados
Venid a mí todos los que estáis trabajados y cargados
 
Bancos
BancosBancos
Bancos
 
экскурсии в алапаевский район (боярская а.)
экскурсии в алапаевский район (боярская а.)экскурсии в алапаевский район (боярская а.)
экскурсии в алапаевский район (боярская а.)
 
Отдельные улицы и кварталы
Отдельные улицы и кварталыОтдельные улицы и кварталы
Отдельные улицы и кварталы
 
Educ.ar tutorial slide share
Educ.ar   tutorial slide shareEduc.ar   tutorial slide share
Educ.ar tutorial slide share
 
ダンバー数150人の法則でFacebookの個人を活用する秘訣
ダンバー数150人の法則でFacebookの個人を活用する秘訣ダンバー数150人の法則でFacebookの個人を活用する秘訣
ダンバー数150人の法則でFacebookの個人を活用する秘訣
 
Clima e tempo cap 14 15 16
Clima e tempo cap 14 15 16Clima e tempo cap 14 15 16
Clima e tempo cap 14 15 16
 
Movimentossaldos
MovimentossaldosMovimentossaldos
Movimentossaldos
 
Você é preconceituoso? Descubra com essa série fotográfica de Joel Pares
Você é preconceituoso? Descubra com essa série fotográfica de Joel ParesVocê é preconceituoso? Descubra com essa série fotográfica de Joel Pares
Você é preconceituoso? Descubra com essa série fotográfica de Joel Pares
 
De teamrollen van Belbin in makkelijk te onthouden steekwoorden
De teamrollen van Belbin in makkelijk te onthouden steekwoordenDe teamrollen van Belbin in makkelijk te onthouden steekwoorden
De teamrollen van Belbin in makkelijk te onthouden steekwoorden
 
Management by Objectives
Management by ObjectivesManagement by Objectives
Management by Objectives
 
Cap 4 5 6
Cap 4  5 6Cap 4  5 6
Cap 4 5 6
 
LTTng-UST: Efficient System-Wide User-Space Tracing
LTTng-UST: Efficient System-Wide User-Space TracingLTTng-UST: Efficient System-Wide User-Space Tracing
LTTng-UST: Efficient System-Wide User-Space Tracing
 
Tema 9 la segunda república española (1931 1936) (1)
Tema 9 la segunda república española (1931 1936) (1)Tema 9 la segunda república española (1931 1936) (1)
Tema 9 la segunda república española (1931 1936) (1)
 

Similar a presentation

Analysis of Multicore Performance Degradation of Scientific Applications
Analysis of Multicore Performance Degradation of Scientific ApplicationsAnalysis of Multicore Performance Degradation of Scientific Applications
Analysis of Multicore Performance Degradation of Scientific ApplicationsJames McGalliard
 
BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!Linaro
 
04536342
0453634204536342
04536342fidan78
 
an introduction lecture to real time operating system
an introduction lecture to real time operating systeman introduction lecture to real time operating system
an introduction lecture to real time operating systemMohammedBelkheiri2
 
Unifying Network Filtering Rules for the Linux Kernel with eBPF
Unifying Network Filtering Rules for the Linux Kernel with eBPFUnifying Network Filtering Rules for the Linux Kernel with eBPF
Unifying Network Filtering Rules for the Linux Kernel with eBPFNetronome
 
eBPF - Observability In Deep
eBPF - Observability In DeepeBPF - Observability In Deep
eBPF - Observability In DeepMydbops
 
Evaluation of Data Reliability on Linux File Systems
Evaluation of Data Reliability on Linux File SystemsEvaluation of Data Reliability on Linux File Systems
Evaluation of Data Reliability on Linux File SystemsYoshitake Kobayashi
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Using BigBench to compare Hive and Spark (short version)
Using BigBench to compare Hive and Spark (short version)Using BigBench to compare Hive and Spark (short version)
Using BigBench to compare Hive and Spark (short version)Nicolas Poggi
 
詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systemshdhappy001
 
詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systemshdhappy001
 
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2Tyrone Systems
 
ITC 360Professor John CovingtonSystem Administration And Managemen.docx
ITC 360Professor John CovingtonSystem Administration And Managemen.docxITC 360Professor John CovingtonSystem Administration And Managemen.docx
ITC 360Professor John CovingtonSystem Administration And Managemen.docxchristiandean12115
 
ICDE2010 Nb-GCLOCK
ICDE2010 Nb-GCLOCKICDE2010 Nb-GCLOCK
ICDE2010 Nb-GCLOCKMakoto Yui
 
Intel new processors
Intel new processorsIntel new processors
Intel new processorszaid_b
 
Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overviewLinaro
 
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]RootedCON
 

Similar a presentation (20)

Analysis of Multicore Performance Degradation of Scientific Applications
Analysis of Multicore Performance Degradation of Scientific ApplicationsAnalysis of Multicore Performance Degradation of Scientific Applications
Analysis of Multicore Performance Degradation of Scientific Applications
 
BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!
 
04536342
0453634204536342
04536342
 
an introduction lecture to real time operating system
an introduction lecture to real time operating systeman introduction lecture to real time operating system
an introduction lecture to real time operating system
 
Unifying Network Filtering Rules for the Linux Kernel with eBPF
Unifying Network Filtering Rules for the Linux Kernel with eBPFUnifying Network Filtering Rules for the Linux Kernel with eBPF
Unifying Network Filtering Rules for the Linux Kernel with eBPF
 
eBPF - Observability In Deep
eBPF - Observability In DeepeBPF - Observability In Deep
eBPF - Observability In Deep
 
Evaluation of Data Reliability on Linux File Systems
Evaluation of Data Reliability on Linux File SystemsEvaluation of Data Reliability on Linux File Systems
Evaluation of Data Reliability on Linux File Systems
 
Packet sniffing
Packet sniffingPacket sniffing
Packet sniffing
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Using BigBench to compare Hive and Spark (short version)
Using BigBench to compare Hive and Spark (short version)Using BigBench to compare Hive and Spark (short version)
Using BigBench to compare Hive and Spark (short version)
 
詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems
 
詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems詹剑锋:Big databench—benchmarking big data systems
詹剑锋:Big databench—benchmarking big data systems
 
Refining Linux
Refining LinuxRefining Linux
Refining Linux
 
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2
Explore Deep Learning Architecture using Tensorflow 2.0 now! Part 2
 
ITC 360Professor John CovingtonSystem Administration And Managemen.docx
ITC 360Professor John CovingtonSystem Administration And Managemen.docxITC 360Professor John CovingtonSystem Administration And Managemen.docx
ITC 360Professor John CovingtonSystem Administration And Managemen.docx
 
ICDE2010 Nb-GCLOCK
ICDE2010 Nb-GCLOCKICDE2010 Nb-GCLOCK
ICDE2010 Nb-GCLOCK
 
Intel new processors
Intel new processorsIntel new processors
Intel new processors
 
Linux kernel architecture
Linux kernel architectureLinux kernel architecture
Linux kernel architecture
 
Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overview
 
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]
Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]
 

presentation

Notas del editor

  1. Ftrace: global clock
  2. Perf: difference: number of instructions (and L1 misses)
  3. Fast, middle, slow path of LTTng - Write - Switch page - Switch subbuffer (subbuf aligned to page size) Ftrace: - Write - Switch page (event can't be bigger than a page)
  4. - Extra work by the kernel - Extra work by LTTng, test the arguments, etc.
  5. Extrae: fast, but not flexible Event: 2 fields 1st field: event type 2nd field: entering or leaving LTTng signal-safe reentrant, atomic instructions for per-cpu variables
  6. Extrae: doesn't seem to be very optimised Not to take away anything from Extrae, but it seems that not much effort was put into optimizing the fast path
  7. - One of those tracers is not scalable - Many optimisations to be made for LTTng (get CPU id, restartable sequences for critical sections)