Ringmap is a complete FreeBSD packet capturing stack specialized for very high-speed networks. Conventional packet capturing software often suffers packet loss at high data rates close to 1Gb/sec or greater. Ringmap takes significant measures to minimize both packet loss and overall system load during packet capture. The goal of this project is to further develop the new packet capturing software for FreeBSD and integrate it with the generic network drivers and libpcap.
1. Outline
Motivation
Background
Solution
Performance evaluation
Summary
Ringmap Capturing Stack for High Performance
Packet Capturing in FreeBSD
Alexander Fiveg
afiveg@freebsd.org
September 28, 2010
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
2. Outline
Motivation
Background
Solution
Performance evaluation
Summary
Outline
1 Motivation
2 Background
3 Solution
4 Performance evaluation
5 Summary
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
3. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
Motivation
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
4. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
Problem
Problems arising during packet capturing:
High bit rates and packet rates
⇒ high CPU load and packet loss
100 100
Packet loss (%)
CPU usage (%)
80 80
60 60
40 40
20 20
0 0
100 200 300 400 500 600 100 200 300 400 500 600
Bit-Rate (MBit/s) Bit-Rate (MBit/s)
Figure: Capturing 64-Bytes packets. (Bezier curves)
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
5. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
The reason of the Problems
Inefficiency of standard capturing software
To many “expensive” operations in terms of CPU cycles:
System calls
Packet copy operations
Memory allocations
etc. . .
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
6. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
Goal of the project
Increase the capturing performance in FreeBSD
Design and implementation the new capturing software to
minimize the packet loss and CPU load during capturing.
Supported Hardware:
The following Intel GbE Controllers:
8254x, 8257x, 8259x
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
7. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
Approach
Eliminating the packet copy operations
by using shared memory buffers (memory mapping)
Eliminating the memory allocations
by using ring buffers
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
8. Outline
Motivation
Background Problem
Solution Goal of the Project
Performance evaluation
Summary
Approach
Eliminating the packet copy operations
by using shared memory buffers (memory mapping)
Eliminating the memory allocations
by using ring buffers
⇒ ring + mapping = ringmap
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
9. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
Background
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
10. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
Packet Capturing: Hardware View
1 Receiving network packets
CPU’s
receive at network adapter
DMA transfer in RAM
2
2 Filtering the received 3 mass storage
packets RAM
Berkeley Packet Filter
1
3 Storing to the hard disk
using a system call
Network adapter
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
11. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Packet Capturing Stack
Network driver
Receiving packets
Berkley Packet Filter (BPF)
Filtering packets
User-space application
Accessing received packets
Initiates:
Storing packets to hard drive
Output packets information to the terminal
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
12. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
How does standard packet capturing
work
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
13. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
Packet is received and saved
in Adapter-FIFO
Adapter-FIFO Network adapter
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
14. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
DMA-Transfer
Paket-Buffer
Packet is received and saved
in Adapter-FIFO
1
Adapter-FIFO Network adapter
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
15. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
Interrupt
DMA-Transfer
Paket-Buffer
Packet is received and saved
in Adapter-FIFO
1
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
16. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
Interrupt Service Routine
Interrupt
Kernel-Thread
DMA-Transfer
Packet-Buffer
Packet is received and saved xN
in Adapter-FIFO Network driver
ISR
1
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
17. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
Userspace
Packet filtering Kernelspace
BPF-Buffer BPF
Interrupt Service Routine
Interrupt
2 Kernel-Thread
DMA-Transfer
Packet-Buffer
Packet is received and saved xN
in Adapter-FIFO Network driver
ISR
1
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
18. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
FreeBSD Standard Packet Capturing Stack
User-Application
User-Buffer
Access packets Userspace
Libpcap-Functions
Packet filtering 3
Kernelspace
BPF-Buffer BPF
Interrupt Service Routine
Interrupt
2 Kernel-Thread
DMA-Transfer
Packet-Buffer
Packet is received and saved xN
in Adapter-FIFO Network driver
ISR
1
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
19. Outline
Motivation
Packet Capturing
Background
How does it work
Solution
Disadvantages
Performance evaluation
Summary
Disadvantages of Standard Packet Capturing Stack
Three copies per packet
1 DMA: FIFO ⇒ Packet Buffer
2 Packet Buffer ⇒ BPF Buffer
3 BPF Buffer ⇒ Userspace Buffer (*)
Memory allocations
For each new received packet will be a new mbuf allocated
System calls
User-space application receives the packets using read(2)(*)
Saving packets to the hard disk
(*) Using Zero-Copy BPF eliminates the last copy and system call
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
20. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
Solution
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
21. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
Ringmap Packet Capturing Stack
Ringmap: The new packet capturing stack
based on standard FreeBSD packet capturing stack:
based on generic network drivers and libpcap
but network driver and libpcap require small modifications
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
22. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
What did we change in generic
User-Application
Libpcap-Functions
Userspace
Kernelspace
BPF
TCP/IP-Stack
Kernel-Thread
xN
ISR
Network driver Software
Network adapter Interrupt Hardware
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
23. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
What did we change in generic
User-Application
Libpcap-Functions
Userspace
Disabled TCP/IP-Stack
Kernelspace
Only packet capturing BPF
TCP/IP-Stack
Kernel-Thread
xN
ISR
Network driver Software
Network adapter Interrupt Hardware
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
24. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
What did we change in generic
User-Application
Libpcap-Functions
Userspace
Disabled TCP/IP-Stack
Kernelspace
Only packet capturing BPF
BPF is accessible in both TCP/IP-Stack
kernel and user-space Kernel-Thread
xN
ISR
Network driver Software
Network adapter Interrupt Hardware
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
25. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
What did we change in generic
User-Application
Libpcap-Functions
Userspace
Disabled TCP/IP-Stack
Kernelspace
Only packet capturing BPF
BPF is accessible in both TCP/IP-Stack
kernel and user-space Kernel-Thread
xN
Network driver and Libpcap ISR
is modified Network driver Software
BPF is unchanged Network adapter Interrupt Hardware
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
26. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
27. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
User-Application
User-Buffer
generic Userspace
Libpcap-Functions
Kernelspace
BPF-Buffer BPF
Kernel-Thread
Packet-Buffer
xN
ISR
Network driver
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
28. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
User-Application
User-Buffer
generic Userspace
Libpcap-Functions
Eliminate packet copies and Kernelspace
syscalls BPF-Buffer BPF
Kernel-Thread
Packet-Buffer
xN
ISR
Network driver
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
29. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
User-Application
User-Buffer
generic Libpcap-Functions
Eliminate packet copies and Userspace
syscalls BPF-Buffer
Kernelspace
BPF
BPF-Buffer is then not
necessary
Kernel-Thread
Packet filtering is possible
in both kernel and Packet-Buffer
xN
user-space
ISR
Network driver
Adapter-FIFO Network adapter Interrupt
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
30. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
User-Application
User-Buffer
generic Libpcap-Functions
Eliminate packet copies and Userspace
syscalls Kernelspace
BPF
BPF-Buffer is then not
necessary
Kernel-Thread
Packet filtering is possible
in both kernel and Packet-Buffer
xN
user-space
ISR
Mapping the Packet-Buffer Network driver
to user-space Network adapter Interrupt
Adapter-FIFO
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
31. Outline
Motivation
Ringmap Capturing Stack
Background
From generic to ringmap
Solution
Overview
Performance evaluation
Summary
From generic to ringmap
User-Applications
Packet-Buffer
generic Libpcap-Functions
Eliminate packet copies and Userspace
syscalls Kernelspace
BPF
BPF-Buffer is then not
necessary
Kernel-Thread
Packet filtering is possible
in both kernel and Packet-Buffer
xN
user-space
ISR
Mapping the Packet-Buffer Network driver
to user-space Network adapter Interrupt
Adapter-FIFO
⇒ ringmap
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
33. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Measurements and performance
evaluation
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
34. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Goal of measurements
Conducting performance evaluations of ringmap capturing
stack
CPU-Load and packet loss during capturing
Benchmarking: generic vs. ringmap
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
35. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Testbed
Capturer Packet generator
FreeBSD Linux-SMP
control traffic
Linux
generated packets
Test control system
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
36. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Measurement setup
Traffic generation
Linux Kernel Packet Generator (pktgen)
generates network packets at very high speed with different:
packet sizes
bit-rate and packet-rate
Capturing
ringmap and generic stacks
Libpcap-application
accessing packets through call pcap loop()
counting the captured packets
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
37. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Calculation of results
System load
Percentage proportion of time the CPU spent in system mode
Interrupt load is not considered
because of interrupt-throttling it is always constant for
generic and ringmap
syst = 1 − intr − user − nice − idle
Packet loss
Difference between generated and captured packets
Packetsloss = Packetssend − Packetsreceived
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
38. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Testing sequence
1 Login to the capturer to
start of capturing
start of system load measurement applications
2 Login to the Packet-Generator to
generate traffic with:
a certain number of packets
a certain bit-rate and packet-rate
3 Login to the capturer to
stop capturing- and measurement-applications
4 Storing of measured values
Each experiment is repeated five times
The mean and standard deviation is calculated
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
39. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
Results
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
40. Outline
Motivation
Measurement setup
Background
Testing sequence
Solution
Results
Performance evaluation
Summary
generic vs. ringmap: Packet loss
100
64-bytes packets ringmap
200-bytes packets ringmap
300-bytes packets ringmap
64-bytes packets generic
200-bytes packets generic
80 300-bytes packets generic
Standard deviation
60
packet loss (%)
40
20
0
0 100 200 300 400 500 600 700 800 900 1000
Bit-Rate (MBit/s)
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
42. Outline
Motivation
Background Achieved Goals
Solution Future Works
Performance evaluation
Summary
Summary
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
43. Outline
Motivation
Background Achieved Goals
Solution Future Works
Performance evaluation
Summary
Achieved goals
Enhanced Capturing-Performance
very low system load (below 12%) and very low packet loss
(below 0.02%)
only during capturing smallest 64-Bytes packets
for maximal reached packet-rate (> 450000pkts/sec)
Stability and usability of implemented software
during all experiments, no kernel panics and no segmentation
faults occurred
very simple install and remove
two Shell-Scripts to installing and removing the ringmap
Capturing Stacks
Libpcap-applications don’t require modification in order to run
with the ringmap
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
44. Outline
Motivation
Background Achieved Goals
Solution Future Works
Performance evaluation
Summary
Achieved goals 2
Ringmap is easily portable to the other network
controllers
The software contains hardware dependent and hardware
independent parts. Only hardware dependent part require
modifications.
The generic driver and libpcap should contain a few hooks for
calling ringmap-functions.
Packet Filtering
Packet filtering can be accomplished using both libpcap- and
kernel-BPF.
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
45. Outline
Motivation
Background Achieved Goals
Solution Future Works
Performance evaluation
Summary
Achieved goals 3
Multithreaded Capturing
Multiple applications can capture from the same interface.
Partly ported to the 10GbE controller
Currently only one queue is used while capturing. The work on
supporting multiqueue is in progress.
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur
46. Outline
Motivation
Background Achieved Goals
Solution Future Works
Performance evaluation
Summary
Future works
Benchmarking: Zero-Copy BPF vs. ringmap
Support for hardware time stamping
Writing the packets to the disc from within the kernel
Enabling TCP/IP stack
10-GbE-Packet-Capturing:
Multiqueue support
Support for hardware packet filtering
Extending ringmap for packet transmission
Alexander Fiveg Ringmap Capturing Stack for High Performance Packet Captur