Session ID: SFO17-209 Session Name: Realtime Performance of Networking Stack - SFO17-209 Speaker: Ivan Khoronzhuk - Pekka Varis Track: LNG ★ Session Summary ★ Realtime applications have traditionally used interrupt latency as the performance metric. Many industrial , automotive and audio/video applications require deterministic performance from networking. Fixed latency can often be dealt with but jitter is especially problematic. New Ethernet features such as time sensitive networking (TSN) ensure single digit microsecond level jitter in a LAN. The networking driver and stack jitter measurements are presented and analyzed. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/sfo17/sfo17-209/ Presentation: Video: --------------------------------------------------- ★ Event Details ★ Linaro Connect San Francisco 2017 (SFO17) 25-29 September 2017 Hyatt Regency San Francisco Airport --------------------------------------------------- Keyword: http://www.linaro.org http://connect.linaro.org --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://twitter.com/linaroorg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961

1. SFO17-209 - Realtime Performance
of Networking Stack
Pekka Varis (Texas Instruments)
Ivan Khoronzhuk (TI / GlobalLogic), most of the work
Ilias Apalodimas (Linaro), measurements on Armada

2. ENGINEERS AND DEVICES
WORKING TOGETHER
Why is Deterministic Networking Needed?
● Isochronous Control Loop
● For correctness application A needs to talk application B in a bounded amount
of time over Ethernet

3. ENGINEERS AND DEVICES
WORKING TOGETHER
Networking Stack Jitter

4. ENGINEERS AND DEVICES
WORKING TOGETHERhttps://projects.linaro.org/browse/LNG-387
Measure Networking Driver and Stack Latency
and Jitter
Use PTP packets because of
HW support to include
timestamps.

5. ENGINEERS AND DEVICES
WORKING TOGETHER
Tested Hardware
AM572x Sitara SoC board:
- TI -rt kernel 4.9, rt SDK
- applied one fix to allow sw timestamping
- Driver: cpsw
- CPU freq used: 1GHz (supports 1.5GHz)
- CPU num: 2
- CPU type: ARM Cortex-A15
- 1Gbps Ethernet
Marvell Armada 8040
- Linux-4.4.52-armada-17.08
- Driver: mvpp2x
- CPU freq used: 1GHz (supports 2GHz)
- CPU num: 4
- CPU type: ARM Cortex-A72
- 10Gbps Ethernet

6. ENGINEERS AND DEVICES
WORKING TOGETHER
Latency Plot per Packet TX
https://projects.linaro.org/browse/LNG-394
:~# cpuloadgen cpu0=100&
:~# phc2sys -s CLOCK_REALTIME -c eth0 -m -O 0
> /dev/null &
:~# echo 2 > /proc/irq/349/smp_affinity; echo 2 >
/proc/irq/350/smp_affinity
:~# ./plget -i eth0 -u 319 -m tx-lat -n 100000 -s 100
-t > 34_am572_rt_tx_ptp_load0_prio_affpin1.txt &
:~# taskset -cp 1 $(pidof plget)
:~# chrt -f -p 99 $(pidof plget)
packet num: 100000; packet size: 86; pps: 100;
latency, us complete
maximum 82.70
minimum 34.90
mean +- RMS 36.52 ±1.67

7. ENGINEERS AND DEVICES
WORKING TOGETHER
Latency Plot per Packet RX
:~# cpuloadgen cpu0=100&
:~# phc2sys -s CLOCK_REALTIME -c eth0 -m -O 0
> /dev/null &
:~# echo 2 > /proc/irq/349/smp_affinity; echo 2 >
/proc/irq/350/smp_affinity
:~# ./plget -i eth0 -u 319 -m rx-lat -n 100000 >
33_am572_rt_rx_ptp_load0_prio_affpin1.txt &
:~# taskset -cp 1 $(pidof plget)
:~# chrt -f -p 99 $(pidof plget)
packet num: 100000; packet size: 86; pps: 100;https://projects.linaro.org/browse/LNG-394
latency, us driver stack complete
maximum 70.05 121.67 186.16
minimum 27.97 63.12 92.30
mean +- RMS 37.04 ±2.90 67.41 ±5.73 104.45 ±8.15

8. ENGINEERS AND DEVICES
WORKING TOGETHER
Latency Plot per Packet RX (Armada 8040)
:~# ./plget -i eth0 -u 380 -m rx-lat -n 100000 -t >
macchiato_rx_udp_load_prio_noaff.txt &
packet num: 100000; packet size: 512; pps: 100;
latency, us driver stack complete
maximum - 897.68 -
minimum - 9.80 -
mean +- RMS - 110.33 ±19.95 -
For reference, goal is to clarify jitter is present on all HW. Issues with core affinity and
support for HW timestamps per packet limited to testing just stack.

9. ENGINEERS AND DEVICES
WORKING TOGETHER
Analysis
IEEE 802 TSN support will require Linux support for:
● Timing over packet, seems to be well covered in open source
● Configuration of switches and shapers, AVNU seems to address
Applications to leverage single digit jitter is not addressed. Tens of microseconds
of jitter in both transmit and receive.
measurements with different tools ftrace, systemtap, to verify that the results are
100% accurate
TSN Patches for Intel NIC and Cisco/Henrik

10. Thank You
#SFO17
SFO17 keynotes and videos on: connect.linaro.org
For further information: www.linaro.org