The document summarizes benchmarking methodologies and tools for network function virtualization (NFV). It discusses standards from groups like ETSI and IETF, as well as open source projects from OPNFV. Traditional approaches to benchmarking physical network devices need to evolve to address virtual network functions (VNFs) and their supporting infrastructure. New methods are needed to test virtual switches, controllers, and other infrastructure components in NFV environments.

1. June 20–23, 2016 | Berlin, Germany

2. The Hitchhiker / Hacker’s
guide to NFV Benchmarking
Al Morton, Trevor Cooper and Maryam Tahhan

3. Introduction
The landscape of emerging NFV
benchmarking can be very
confusing to navigate.
Goal is to explain the history and
state-of-art NFV benchmarking in
ETSI NFV, IETF and OPNFV

4. Projects
Yardstick
Verification of the infrastructure compliance when running VNF applications.
Umbrella project for performance testing
Storperf Storage performance testing
VSperf Characterize the performance of virtual switches for Telco NFV use cases.
CPerf Controller performance testing
CSIT
(Continuous
System
Integration
and Testing)
Development of software code for fully automated VPP code testing, functionality,
performance, regression and new functions.
Standards /
specifications
Projects Methodologies Tools Workloads

5. http://cdn.meme.am/instances/250x250/62902253.jpg

6. OPNFV Release B Test Tools
Yardstick
Storeperf CPERF VSPERF QTIP
Standards /
specifications
Projects Methodologies Tools Workloads

7. OPNFV Release B Test Coverage
Standards /
specifications
Projects Methodologies Tools Workloads

8. OPNFV Release B Test Coverage
(ETSI-NFV Domains)
Yardstick, VSPERF, CPERF, STORPERF,
Functest and QTIP
Standards /
specifications
Projects Methodologies Tools Workloads

9. Other Tools: Traffic Generators
Traffic Generator Open Source/Proprietary Hardware/Software
IXIA Proprietary SW/HW
Spirent Proprietary SW/HW
Xena Proprietary SW/HW
DPDK pktgen Open Source SW
Moongen Open Source SW
TREX Open Source SW
pktgen Open Source SW
LAN Tornado RFC 2544 Proprietary SW
Seagull Open Source SW
Standards /
specifications
Projects Methodologies Tools Workloads

10. Other Tools/Testing efforts
Testing Efforts
• CSIT
• “NFV Performance Benchmarking for vCPE” test
• EEMBC creates their own tools for the benchmark specs they develop (Commercial)
• Canonical VNF Performance Interoperability Lab (V-PIL)
SDN Tools:
• Cbench (OS) CBench is a somewhat classic SDN controller benchmark tool. It blasts a controller with OpenFlow packet-in
messages and counts the rate of flow mod messages returned.
• WCBench: WCBench consumes CBench as a library, then builds a robust test automation, stats collection and stats
analysis/graphing system around it.
• ClassBench-ng: Test utility for generation of firewall/OpenFlow rules based on original (no longer maintained) Classbench.
• NSTAT (OS): Goes beyond with Multi-Threaded Cbench, adds monitoring OF traffic, switch and controller scale testing.
• Veryx Technologies: Commercial tool, measures metrics in IETF Draft
• EXFO: Commercial tools, Load testing of IMS and EPC virtual network functions (VNF
• Rally, Netperf (TCP_CRR Test), iperf.
• ODL : OpenFlow protocol performance measurements in OpenDaylight.
Standards /
specifications
Projects Methodologies Tools Workloads

11. Workloads
VNF Use Case Brief Description Pros Cons Details
Clear Water
IMS
IMS Control Plane Clear Water is an open source
implementation of IMS (the IP
Multimedia Subsystem) designed
to provide voice, video and
messaging services. Clear water
handles only control plane traffic
for sessions.
Open source, All in
one and Scalable
VNF
Available in AMI and OVI
formats, Supports Openstack,
All In one Install is limited to
fixed domains and is
recommended not to use
this, Multi-instance VNF
Image requires Chef
integration, however, scripts
are provided by Clear Water
for IMS deployment
IMF Use Case with Clear
Water
vFW Virtual Firewall (canonical
VNF)
Open Source (GNU
GPL)
IPTables
vAAA Virtual AAA (canonical
VNF)
Open Source (BSD) Free Radius
Emerginov IMS control planned - like integration of open source
solutions (HTTP/SIP proxies, Media
serveur,..) managed by Puppet
Open Source Emerginov
vRouter/vCPE OpenWRT OpenWRT is frequently used as
virtual CPE
Open Source OpenWRT images for
OpenStack
vIDS Snort Intrusion Detection System Open Source www.snort.org
Traffic Control vCDN Content Delivery Network Open Source Traffic Control on github
OpenSIPS vSBC, SIP server Open SIP Server, which is adaptible
and can be used as a vSBC
OpenSIPS website
Standards /
specifications
Projects Methodologies Tools Workloads

12. What is Benchmarking?

13. Benchmark Definition in IETF
A standardized test that serves as a basis for performance evaluation and
comparison.
13
STANDARDIZED
• All aspects of the
testing conditions
are sufficiently
specified and
controlled such that
the results:
• Are informative.
• Can be reported
unambiguously.
• Are repeatable.
BASIS FOR
PERFORMANCE
EVALUATION
• The tests measure
some performance
characteristic that:
• Is relevant and
meaningful to
users.
• Provides insight to
improve an inter-
networking
technology.
BASIS FOR
COMPARISON
• The tests can be:
• Performed on
different vendors'
devices without
modification.
• Observed using
signals that are
external to the
device/system.
Benchmarks are NOT
• Functional tests.
• PASS/FAIL criteria.
• Performed on live
networks, or
performed with live
network traffic.

14. http://memberfiles.freewebs.com/94/67/101646794/photos/undefined/E_Marvin%20The%20Paranoid%20Android(THGTTG_Garth%20Jennings_2005)-3.jpg

15. NFV Benchmarking Landscape
Standards /
specifications
• IETF
• ETSI NFV
• Other SDOs/Open
Source projects
OPNFV Projects
• Yardstick
• VSPERF
• CPERF
• STORPERF
Methodologies
• Traditional PNF
benchmarking
• Evolving methods
to Benchmark
Virtual Network
Functions
• Benchmarks of
new Infrastructure
Components
Tools
• Test Frameworks
• Traffic generators
Workloads
• Stimulus
• Compute/Memory
• VNFs
Let’s break it down….

16. Standards / Specifications
SDOs
BMWG
Internet drafts
RFCs
Service Quality Metrics
TST WG WIs
Other SDOs/Open Source
projects
IEEE
EEMBC (Cloud DC,
Streaming/Web, Bug Data)
Standards /
specifications
Projects Methodologies Tools Workloads
Other
SDOs/Open
Source
projects

17. BMWG, since 1989
RFC 1242 (was draft-ietf-bmwg-terms) Benchmarking Terminology for Network Interconnection Devices
RFC 1944 (was draft-ietf-bmwg-methodology) Benchmarking Methodology for Network Interconnect Devices
RFC 2285 (was draft-ietf-bmwg-lanswitch) Benchmarking Terminology for LAN Switching Devices
RFC 2432 (was draft-ietf-bmwg-mcast)Terminology for IP Multicast Benchmarking
RFC 2647 (was draft-ietf-bmwg-secperf) Benchmarking Terminology for Firewall Performance
RFC 2761 (was draft-ietf-bmwg-atm-term) Terminology for ATM Benchmarking
RFC 2889 (was draft-ietf-bmwg-mswitch) Benchmarking Methodology for LAN Switching Devices
RFC 3116 (was draft-ietf-bmwg-atm-method) Methodology for ATM Benchmarking
RFC 3133 (was draft-ietf-bmwg-fr-term) Terminology for Frame Relay Benchmarking
RFC 3134 (was draft-ietf-bmwg-atm-term-abr) Terminology for ATM ABR Benchmarking
RFC 3222 (was draft-ietf-bmwg-fib-term) Terminology for Forwarding Information Base (FIB) based Router Performance
RFC 3511 (was draft-ietf-bmwg-firewall) Benchmarking Methodology for Firewall Performance
RFC 3918 (was draft-ietf-bmwg-mcastm)Methodology for IP Multicast Benchmarking
RFC 4061 (was draft-ietf-bmwg-ospfconv-intraarea) Benchmarking Basic OSPF Single Router Control Plane
RFC 4062 (was draft-ietf-bmwg-ospfconv-term) OSPF Benchmarking Terminology and Concepts
RFC 4063 (was draft-ietf-bmwg-ospfconv-applicability) Considerations When Using Basic OSPF Convergence Benchmarks
RFC 4098 (was draft-ietf-bmwg-conterm) Terminology for Benchmarking BGP Device Convergence in the Control Plane
RFC 4689 (was draft-ietf-bmwg-dsmterm) Terminology for Benchmarking Network-layer Traffic Control Mechanisms
RFC 4814 (was draft-ietf-bmwg-hash-stuffing) Hash and Stuffing: Overlooked Factors in Network Device Benchmarking
RFC 4883 (was draft-ietf-bmwg-benchres-term) Benchmarking Terminology for Resource Reservation Capable Routers
RFC 5180 (was draft-ietf-bmwg-ipv6-meth) IPv6 Benchmarking Methodology for Network Interconnect Devices Errata
RFC 5695 (was draft-ietf-bmwg-mpls-forwarding-meth) MPLS Forwarding Benchmarking Methodology for IP Flows Errata
RFC 6201 (was draft-ietf-bmwg-reset) Device Reset Characterization
RFC 6412 (was draft-ietf-bmwg-igp-dataplane-conv-term) Terminology for Benchmarking Link-State IGP Data-Plane Route Convergence
RFC 6413 (was draft-ietf-bmwg-igp-dataplane-conv-meth) Benchmarking Methodology for Link-State IGP Data-Plane Route Convergence
RFC 6645 (was draft-ietf-bmwg-ipflow-meth) IP Flow Information Accounting and Export Benchmarking Methodology
RFC 6815 (was draft-ietf-bmwg-2544-as) Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful
RFC 6894 (was draft-ietf-bmwg-protection-meth) Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute Protection
RFC 6985 (was draft-ietf-bmwg-imix-genome) IMIX Genome: Specification of Variable Packet Sizes for Additional Testing
RFC 7501 (was draft-ietf-bmwg-sip-bench-term)Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration
RFC 7502 (was draft-ietf-bmwg-sip-bench-meth)Methodology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration
RFC 7640 (was draft-ietf-bmwg-traffic-management) Traffic Management Benchmarking
RFC 7654 (was draft-ietf-bmwg-issu-meth) Benchmarking Methodology for In-Service Software Upgrade (ISSU)
RFC 7747 (was draft-ietf-bmwg-bgp-basic-convergence) Basic BGP Convergence Benchmarking Methodology for Data-Plane Convergence
Standards /
specifications
Projects Methodologies Tools Workloads

18. BMWG Products in Summary
• RFC 1242 and RFC 2544, Throughput and Latency
• LAN Switches, Firewalls, Multicast
• ISIS, OSPF, and BGP Convergence
• MPLS, MPLS-TE FRR, Flow Export, IPv6
• RFC 2544 Applicability Statement
• IMIX Genome
• SIP Device Benchmarking
• Traffic Management features
• In-Service Software Upgrade
• Basic BGP Convergence

19. BMWG NFV Internet Drafts
Internet Draft Link
Data Centre Benchmarking Methodology draft-ietf-bmwg-dcbench-methodology-02
Data Centre Benchmarking Terminology draft-ietf-bmwg-dcbench-terminology-05
Benchmarking Methodology for SDN Controller
Performance
draft-ietf-bmwg-sdn-controller-benchmark-meth-01
Terminology for Benchmarking SDN Controller
Performance
draft-ietf-bmwg-sdn-controller-benchmark-term-01
Considerations for Benchmarking Virtual Network
Functions and Their Infrastructure
draft-ietf-bmwg-virtual-net-02
Considerations for Benchmarking High Availability of
NFV Infrastructure
draft-kim-bmwg-ha-nfvi-01
Benchmarking Methodology for EVPN draft-kishjac-bmwg-evpntest-00
VNF Benchmarking Methodology draft-rosa-bmwg-vnfbench-00
Benchmarking Virtual Switches in OPNFV draft-vsperf-bmwg-vswitch-opnfv-02
Standards /
specifications
Projects Methodologies Tools Workloads

20. ETSI NFV Testing Specifications I
• Specifies performance benchmarking metrics for virtual switching
• References: IETF RFC 2544, : IETF RFC 2679, : IETF RFC 2680, : IETF RFC 3511, : IETF RFC 4737, :
IETF RFC5481, IETF RFC 6703, IETF RFC 6049, IETF RFC 7348, IETF RFC 6815, IETF RFC 6985,
draft-ietf-bmwg-ipsec-term-12.txt, ipsec-meth-12.txt, draft-ietf-bmwg-virtual-net-01.txt, draft-
vsperf-bmwg-vswitch-opnfv-01.txt.
ETSI GS NFV-IFA 003: vSwitch Benchmarking and Acceleration
Specification
• Provides guidelines for test plans that assess different approaches to defining SDN Applications,
different ways of arranging and federating SDN Controllers, and arrangements of network
switching/forwarding functions (both physical and virtual) to create the various path-
implementations between and among NS Endpoints and VNFs
ETSI GS NFV-TST004: Guidelines for Test Plan on Path Implementation
through NFVI (in-progress)
Standards /
specifications
Projects Methodologies Tools Workloads

21. ETSI NFV Testing Specifications II
• Provides a list of features which the performance and portability templates (Virtual Machine
Descriptor and Compute Host Descriptor) should contain for the appropriate deployment of
Virtual Machines over a Compute Host (i.e. a "telco datacentre").
• In addition, the document provides a set of recommendations and best practises on the
minimum requirements that the HW and hypervisor should have for a "telco datacentre"
suitable for data-plane workloads
• References: IEEE 1588, IETF RFC 2544, IETF RFC 2889, IETF RFC 3393, IETF RFC 3511, IETF RFC
3918.
ETSI - GS NFV-PER 001: NFV Performance & Portability Best Practise
• Enumerates metrics for NFV infrastructure, management and orchestration service qualities
that can impact the end user service qualities delivered by VNF instances hosted on NFV
infrastructure.
• References: IETF RFC 2330, IETF RFC 6390, ISO/IEC 15939, ITU-T I.356, ITU-T M.3341, ITU-
T.1543, NIST SP 800-145.
ETSI - GS NFV-INF 010: NFVI Service Quality Metrics
Standards /
specifications
Projects Methodologies Tools Workloads

22. ETSI NFV Testing Specifications III
• Recommendations for lab validation of VNFs, their interaction with the NFV
functional
blocks and the NFV blocks themselves.
ETSI GS NFV-TST001: PRE-DEPLOYMENT TESTING
• Study how interoperability test methodology can be applied to NFV by
analysing the
functional blocks and interfaces defined within the NFV architecture
ETSI GS NFV-TST002: INTEROPERABILITY TESTING
METHODOLOGY
Standards /
specifications
Projects Methodologies Tools Workloads

23. Methodologies
• Traditional: Physical Network Function (PNF)
Benchmarking
• Evolving methods to Benchmark Virtual Network
Functions (VNF)
• Benchmarks of new Infrastructure Components
• SDN Controllers and Virtual Switches
Standards /
specifications
Projects Methodologies Tools Workloads

24. Benchmarking Physical Network Devices
(Functions)
Test
Generator
Test
ReceiverDUT
CONFIG Isolated Test Environ.
Test
Generator
Test
Receiver
Isolated Test Environ.
Easy Calibration and
Resource tracking
• How Fast?
• in Packets or Bytes?
• How much Latency?
• How much Capacity?
• How many Sessions?
• How Quick is Recovery?
and importantly
• How do Vendor A & B Compare?
Standards /
specifications
Projects Methodologies Tools Workloads

25. But can we still apply this methodology in the
context of NFV benchmarking?
Standards /
specifications
Projects Methodologies Tools Workloads

26. Evolving methods to Benchmark Virtual Network
Functions (VNFs)
• System Under Test is different
• Workload = Stimulus and SUT
Standards /
specifications
Projects Methodologies Tools Workloads

27. Benchmarking Virtual Network Functions
(VNF + NFVI)
• SUT Configuration is also different
27
Test
Generator
Test
Receiver
SUT
CONFIG
Isolated Test Environ.
VM - VNF
Virtual
Resources
General-Purpose Compute Platform
CONFIG
CONFIG
Operating System/Other SW CONFIG
Standards /
specifications
Projects Methodologies Tools Workloads

28. Test Configuration Parameters
28
General purpose platform:
• Number of server blades (shelf occupation)
• CPUs/cores
• Storage system
• Memory
• BIOS Configuration
Configurations that support the VNF:
• Hypervisor
• Virtual Machine (and resources allocated to the VM)
• Infrastructure Virtual Network
The VNF itself:
• Specific function being implemented in VNF
• Number of VNF components in the service function chain
• Number of physical interfaces and links transited in the service
function chain
Note: This is a PARTIAL list of items that need to be configured and reported with the test results!
Standards /
specifications
Projects Methodologies Tools Workloads

29. More Benchmarking Considerations
29
• Comparison with Physical Network Functions
– Re-use of existing benchmarks, for a fair comparison.
• Continued Emphasis on Black-Box Benchmarks
– Auxiliary Metrics are tempting
– Supply both, may provide useful insight
• New Benchmarks for a Dynamic World
– Time to deploy VNFs, Time to Migrate, Power
Consumption
• Assessment of Benchmark Coverage

30. Assess Benchmark Coverage:
3 x 3 became 3 x 4 Matrix
30
Criteria
Functions
SPEED ACCURACY RELIABILITY SCALE
Activation /
Creation /
Setup
Operation
De-Activation /
Deletion /
Take-Down

31. SDN Controller Benchmarks
Speed Scalability Reliability
Setup 1. Network Topology Discovery.
2. Reactive Path Provisioning Time.
3. Proactive Path Provisioning Time.
4. Reactive Path Provisioning Rate .
5. Proactive Path Provisioning Rate.
1. Network Discovery Size
Operational 1. Asynchronous Message Processing
Rate.
2. Asynchronous Message Processing
Time.
1. Control Sessions Capacity.
2. Forwarding Table
Capacity.
1. Network Topology
Change Detection Time.
2. Exception Handling.
3. Denial of Service
Handling
4. Network Re-Provisioning
Time
Teardown 1. Controller Failover Time.
31
We now also have to benchmark new Infrastructure components
- SDN Controllers
- Virtual Switches

32. NFV Benchmarking Gaps
• Standards for characterizing Network Services performance in realistic/scaled deployment scenarios
• Consistent and extensible architecture for NFV performance testing including MANO
• Representative Open Source workloads (Compute/memory and Traffic)
• Lifecycle performance metrics and methodologies
Closing the Gaps
• Develop APIs and models for configuration of workloads (Compute/memory and Traffic)
• Automate for testing in diverse environments and deployment scenarios
• Develop Packet generator features for new protocols and traffic profiles
• Unify performance testing frameworks in OPNFV
Reaching our Destination
NetworkServicePerformance
Community test tools and infrastructure
Test Dashboards
Database of test results

33. Summary
There is an ongoing effort to ensure alignment across SDOs and Open Source to help converge
on standardized tests that serve as a basis for NFV performance evaluation and comparison.

34. Questions?
http://fordprefectstowel.weebly.com/uploads/2/5/2/3/25231015/6078343_orig.png

35. http://littlefun.org/uploads/5269efe1e691b24d64aec592_736.jpg

36. Back up

37. References1. http://tolly.com/ts/2011/Blade/G8264/Tolly211108BLADERackSwitchG8264.pdf
2. https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02
3. Intel® Open Network Platform Release 2.0 Performance Test Report (see legal
information p.76) https://01.org/packet-processing/intel%C2%AE-onp
4. https://wiki.opendaylight.org/view/File:Opendaylight-test.pptx
5. https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-term-01
6. http://tools.ietf.org/html/draft-vsperf-bmwg-vswitch-opnfv-02.txt
7. http://bsdrp.net/documentation/technical_docs/performance
8. http://data.guug.de/slides/lk2008/10G_preso_lk2008.pdf
9. http://www.telematica.polito.it/oldsite/courmayeur06/papers/06-A.2.1.pdf
10. https://www.ietf.org/proceedings/89/slides/slides-89-bmwg-6.pdf
11. http://www.lightreading.com/ethernet-ip/40-gig-router-test-results/d/d-id/608230
12. https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test#Act
ively_Used_or_Under_Active_Development
13. http://www.ixiacom.com/sites/default/files/resources/case-study/benchmarking-
opensource-sdn.pdf
14. https://groups.google.com/a/onosproject.org/forum/?hl=en#!topic/onos-co/qGg2LAnUxLA
37

38. Evolving methods to Benchmark Virtual Network
Functions (VNFs)
• System Under Test is different
• Provisioning VNF and Virtual Network
• COTS configuration
• Virtualization layer choices
• Shred resources
• Capacity considerations
• Performance metrics
Standards /
specifications
Projects Methodologies Tools Workloads

39. System Under Test (SUT): NFVI component

40. System Under Test (SUT): VNF

41. System Under Test (SUT): NFVI

42. System Under Test (SUT): VNF + NFVI

43. System Under Test (SUT): MANO

44. System Under Test (SUT): VNF + NFVI +
MANO

45. Other Tools/Testing efforts
• CSIT
• SDN Tools:
• Cbench (OS) CBench is a somewhat classic SDN controller benchmark tool. It blasts a
controller with OpenFlow packet-in messages and counts the rate of flow mod messages
returned.
• WCBench: WCBench consumes CBench as a library, then builds a robust test automation, stats
collection and stats analysis/graphing system around it.
• ClassBench-ng: Test utility for generation of firewall/OpenFlow rules based on original (no longer
maintained) Classbench.
• NSTAT (OS): Goes beyond with Multi-Threaded Cbench, adds monitoring OF traffic, switch and
controller scale testing.
• Veryx Technologies: Commercial tool, measures metrics in IETF Draft
• EXFO: Commercial tools, Load testing of IMS and EPC virtual network functions (VNF
• EEMBC creates their own tools for the benchmark specs they develop (Commercial)
• Rally
• ODL : OpenFlow protocol performance measurements in OpenDaylight.
• Canonical VNF Performance Interoperability Lab (V-PIL)
Standards /
specifications
Projects Methodologies Tools Workloads