Performance Evaluation and Tuning of Virtual Infrastructure Managers for (Micro) Virtual Network Functions Virtualized Network Functions (VNFs) are emerging as the keystone of 5G network architectures: flexibility, agility, fast instantiation times, consolidation, Commercial Off The Shelf (COTS) hardware support and significant cost savings are fundamental for meeting the requirements of the new generation of mobile networks. In this paper we deal with the management of the virtual computing resources for the execution of Micro VNFs. This functionality is performed by the Virtual Infrastructure Manager (VIM) in the NFV MANagement and Orchestration (MANO) reference architecture. We discuss the VIM instantiation process and propose a generic reference model, starting from the analysis of two Open Source VIMs, namely OpenStack Nova and Nomad. We implemented a tuned version of the VIMs with the specific goal of reducing the duration of the instantiation process. We realized a performance comparison of the two VIMs, both considering the plain and the tuned versions. The tuned VIMs and the performance evaluation tools that we have employed are provided openly and can be downloaded from our repository.

1. Performance Evaluation and Tuning of Virtual Infrastructure
Managers for (Micro) Virtual Network Functions
Pier Luigi Ventre(1), Claudio Pisa(2), Stefano Salsano(1,2), Giuseppe Siracusano(1,3),
Florian Schmidt(3), Paolo Lungaroni(4), Nicola Blefari-Melazzi(1,2)
(1)Univ. of Rome Tor Vergata, Italy; (2)CNIT, Italy
(3)NEC Laboratory Europe, Germany; (4)Consortium GARR, Italy
November 8th 2016 – IEEE NFV-SDN conference, Palo Alto, USA, 7-9 Nov. 2016
A super-fluid, cloud-native, converged edge system

2. Outline
• The SUPERFLUIDITY project – goals and approach
– Toward sub 10 ms service instantiation
• A Unikernel primer
– Memory footprint and boot time results
• Orchestration of Unikernels
– Virtual Infrastructure Managers (VIMs): analysis of performance
– Tuning of VIM performance
2

3. SUPERFLUIDITY project - http://superfluidity.eu/
Goal: a superfluid NFV approach
• Instantiate network functions and services on-the-fly
• Run them anywhere in the network (core, aggregation, edge), across
heterogeneous infrastructure environments (computing and networking),
taking advantage of specific hardware features, such as high performance
accelerators, when available
Approach
• Decomposition of network components and services into elementary and
reusable primitives (“Reusable Functional Blocks – RFBs”)
• Platform-independent abstractions, permitting reuse of network functions
across heterogeneous hardware platforms
3

4. SUPERFLUIDITY project - http://superfluidity.eu/
Project consortium
• The SUPERFLUIDITY Consortium includes 18 partners from 12 countries. The
partners include 4 universities, 1 inter-university research consortium, 9
industrial partners and 4 SMEs
Project timeline
• July 2015 – March 2018
Disclaimer
• The work presented here is a (small) subset of the work performed in the
project
4

5. • Classical NFV environments (i.e. by ETSI NFV standards)
– VNFs are composed/orchestrated to realize Network Services
– VNFs can be decomposed in VNFC (VNF Components)
«Big»
VNF
«Big»
VNF
«Big»
VNF
«Big»
VNF
VNF
C
VNF
C
VNF
C
VM
VM
VM
Heterogeneous composition/execution environments
5

6. Heterogeneous composition/execution environments
• Towards more «fine-grained» decomposition…
• E.g. modular software routers (like Click)
– Click elements are combined in configurations (Direct Acyclic Graphs)
• E.g. XSFM-based (eXtended Finite State Machine) decomposition of traffic
forwarding / flow processing tasks, and HW support for wire speed execution
6

7. Towards sub 10 ms service instantiation
Why a superfluid NFV
• Quick provisioning of services: JIT proxies, firewalls, on-the-fly monitoring
• Quick migration of services
• Hosting large number of services on the same server: e.g., vCPE
• Optimized use of resources thanks to dynamic sharing
• General investment and operating cost reductions
7

8. We need a superfluid virtualization : use of Unikernels
Containers
e.g. Docker
• Lightweight
• Poor isolation
8
Hypervisors (traditional VMs)
e.g. XEN, KVM, wmware…
• Strong isolation
• Heavyweight
Unikernels
Specialized VMs (e.g. MiniOS, ClickOS…)
• Strong isolation
• Very Lightweight
• Very good security properties
They break the “myth” of VMs being heavy weight…

9. Outline
• The SUPERFLUIDITY project – goals and approach
– Toward sub 10 ms service instantiation
• A Unikernel primer
– Memory footprint and boot time results
• Orchestration of Unikernels
– Virtual Infrastructure Managers (VIMs): analysis of performance
– Tuning of VIM performance
9

10. A Unikernel Primer
• Specialized VM: single
application + minimalistic OS
• Single address space,
co-operative scheduler so low
overheads
• Unikernel virtualization
platforms extend existing
hypervisors (e.g. XEN)
driver1
driver2
app1
(e.g., Linux, FreeBSD)
KERNELSPACEUSERSPACE
app2
appNdriverN
Vdriver1
vdriver2
app
SINGLEADDRESS
SPACE
10
General purpose OS Unikernel
a minimalistic OS
(e.g., MiniOS, Osv)

11. Example Unikernel Memory Footprint (ClickOS)
• Hello world guest VM
– 296KB
• Ponger (ping responder) guest VM : ~700KB
– 350KB come from lwip and newlibc
– this is with minor optimizations to MiniOS
(e.g., reducing the threads’ stack size)
11

12. Unikernels boot time
• Without xen store: 1.43 ms
Guest configuration: MiniOS, 1 VCPU, 8MB RAM, 1 VIF
• Without libxl: 6.67 msecs
• 87.77 msecs
12
State of the art results
Recent results (from SUPERFLUIDITY),
by redesigning the toolstack

13. Outline
• The SUPERFLUIDITY project – goals and approach
– Toward sub 10 ms service instantiation
• A Unikernel primer
– Memory footprint and boot time results
• Orchestration of Unikernels
– Virtual Infrastructure Managers (VIMs): analysis of performance
– Tuning of VIM performance
13

14. ETSI MANagement and Orchestration (MANO) Model
14

15. VM instantiation and boot time
15
Orchestrator
request

16. VM instantiation and boot time
16
Orchestrator
request
VIM
operations
Virtualization
Platform
Guest OS (VM)
Boot time
1-2 s
5-10 s
~1 s

17. VM instantiation and boot time
17
Orchestrator
request
VIM
operations
Virtualization
Platform
Guest OS (VM)
Boot time
1-2 s
~1 ms
~1 ms
• Unikernels can provide low
latency instantiation times for
“Micro-VNF”
• What about VIMs (Virtual
Infrastructure Managers) ?

18. Performance analysis and Tuning of VIMs for Micro VNFs
• General model of the VNF instantiation process
• Modifications to VIMs to instantiate Micro-VNFs based on
ClickOS Unikernel
• Methodology to evaluate the performances
• Performance Evaluation
18

19. Virtual Infrastructure Managers (VIMs)
We considered the performance of two VIMs :
• OpenStack Nova
– OpenStack is composed by subprojects
– Nova: orchestration and management of computing resources ---> VIM
– 1 Nova node (scheduling) + several compute nodes (which interact with the hypervisor)
– Not tied to a specific virtualization technology
• Nomad by HashiCorp
– Minimalistic cluster manager and job scheduler
– Nomad server (scheduling) + Nomad clients (interact with the hypervisor)
– Not tied to a specific virtualization technology
19

20. Reference Model of the VNF instantiation process
20

21. Mapping of the reference model to the considered VIMs
21

22. VIM instantiation model for Openstack Nova
22

23. VIM instantiation model for nomad
23

24. VIM modifications to instantiate (ClickOS) Micro VNFs
24
A regular VM can boot its OS
from an image or a disk snapshot
that can be read from an
associated block device (disk).
The host hypervisor instructs the
VM to run the boot loader, which
reads the kernel image from the
block device.
ClickOS based MicroVNFs, are
shipped as a tiny kernel without
a block device. These VMs need
to boot from a so-called diskless
image. The host hypervisor reads
the kernel image from a file or a
repository and directly injects it
in the VM memory.
Virtual
Infrastructure
Manager
Virtualization
Platform
(Hypervisor)
This interface needs to
be modified to support
the boot of “diskless
images”

25. VIM modifications to instantiate (ClickOS) Micro VNFs
• OpenStack
– Xen supported out of the box, using the Libvirt toolstack
– We considered the boot of diskless images targeting only one component
(Nova Compute) and a specific toolstack, Libvirt.
– Libvirt talks with Xen using libxl the default Xen toolstack API.
– We modify the XML description of the guest domain provided by the driver,
changing the XML description on the fly before the creation of the VM
• Nomad
– Xen not supported out of the box
– We developed a new Nomad driver for Xen, called XenDriver .
– The new driver communicates with the XL Xen toolstack and it is also able to
instantiate a ClickOS VM.
25

26. VIM performance evaluation approach
• We evaluate the VM scheduling and instantiation phase, combining message trace
analysis and timestamps in the code
• Message traces (coarse information, beginning and end of the different phases)
– VIM Message Analyzer capable of analyzing Nova and Nomad message exchanges
• Detailed breakdown with timestamps in the code (Nomad Client, Nova Compute)
• Workload generators:
– OpenStack : Rally benchmarking tool
– Nomad : developed the “Nomad Pusher”, a utility written in the GO language which
programmatically submits jobs to the Nomad Server.
26

27. Results – ClickOS instantiation times
27
OpenStack Nova
Nomad
seconds
seconds

28. There is no comparison implied…
• NB: the purpose of the work is NOT to compare OpenStack vs. Nomad.
The goal is to understand how both behave and find ways to reduce
instantiation times.
• A direct comparison makes few sense. OpenStack is a much more
complete framework in terms of offered functionality and different
types of supported hypervisors. Moreover, the comparison is unfair
also because for the Nomad case we have developed a driver only
targeted to support the Xen/Click OS case.
28

29. VIM Tuning
• OpenStack
– Diskless VM -> we can skip most of the actions performed during the image creation;
– UniKernels are special purpose VMs:
• SSH is really needed ?
• Full-IP stack ?
– We were able to reduce the spawning time of about 70%
– Looking at the overall instantiation time, the relative reduction is about 45%;
• Nomad
– No much space for the optimization;
• We implemented only the necessary functionality;
– We introduced further improvements assuming a local store for the Micro VNFs,
reducing the Driver operation of about 30 ms;
29

30. seconds
seconds
Results – OpenStack details and tuning
30
OpenStack Nova overall
OpenStack Nova spawn phase

31. Results – Nomad details and tuning
31
Nomad overall
Nomad spawn phase
seconds
seconds

32. VIM performances - Ongoing & Future Work
• Consider the impact of system load on the performance
– Measure the average instantiation times considering batches of incoming requests with given
rate (requests/s) and arrival patterns.
– Analyze the impact of the number of already allocated VMs and of the number of target nodes
to be deployed.
• Keep improving the performance of the considered VIMs
– e.g. trying to replace the lazy notification mechanism of Nomad with a reactive approach
• Extend the analysis to another VIM
– OpenVIM from the OSM project
32

33. Unikernel virtualization in the SUPERFLUIDITY vision
• We have considered the optimization of Unikernel virtualization and the needed
enhancements to Virtual Infrastructure Managers to support Unikernels.
• In the SUPERFLUIDITY vision, Unikernels are interesting as they support the
decomposition of network services in “smaller” components that can be deployed
on the fly (NB: Unikernels are complementary to other approaches!)
• The NFV Infrastructure should be extended in order to support Unikernel
virtualization in addition to traditional VMs. This way it will be possible to design
services that exploit the most efficient solutions depending on several factors.
33

34. Conclusions
• Unikernel virtualization can provide VM instantiation and boot time in
the order of ms
– ongoing: consolidation of results, generic and automatic optimization process for
hypervisor toolstack and for guests
• Work is still needed at the level of Virtual Infrastructure Managers
– e.g. OpenStack (~ 1 s), Nomad (~ 300 ms)
• VIMs are currently designed for generality, the challenge is to specialize
them in a flexible way, keeping the compatibility with the mainstream
versions
34

35. References & paper download
• SUPERFLUIDITY project Home Page http://superfluidity.eu/
• G. Bianchi, et al. “Superfluidity: a flexible functional architecture for 5G
networks”, Transactions on Emerging Telecommunications Technologies 27, no.
9, Sep 2016
• P. L. Ventre, C. Pisa, S. Salsano, G. Siracusano, F. Schmidt, P. Lungaroni,
N. Blefari-Melazzi, “Performance Evaluation and Tuning of Virtual Infrastructure
Managers for (Micro) Virtual Network Functions”,
IEEE NFV-SDN Conference, Palo Alto, USA, 7-9 November 2016
http://netgroup.uniroma2.it/Stefano_Salsano/papers/salsano-ieee-nfv-sdn-2016-vim-performance-for-unikernels.pdf
35

36. References – Speed up of Virtualization Platforms / Guests
• J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco, F. Huici,
“ClickOS and the art of network function virtualization”, NSDI 2014, 11th
USENIX Conference on Networked Systems Design and Implementation,
2014.
• F. Manco, J. Martins, K. Yasukata, J. Mendes, S. Kuenzer, F. Huici,
“The Case for the Superfluid Cloud”, 7th USENIX Workshop on Hot Topics in
Cloud Computing (HotCloud 15), 2015
• Recent unpublished results are included in this presentation:
S. Salsano, F. Huici, “Superfluid NFV: VMs and Virtual Infrastructure
Managers speed-up for instantaneous service instantiation”, invited talk at
EWSDN 2016 workshop, 10 October 2016, The Hague, Netherlands
http://www.slideshare.net/stefanosalsano/superfluid-nfv-vms-and-virtual-infrastructure-managers-speedup-for-instantaneous-service-instantiation
36

37. Thank you. Questions?
Contacts
Stefano Salsano
University of Rome Tor Vergata / CNIT
stefano.salsano@uniroma2.it
The tools we developed are available on github
https://github.com/netgroup/vim-tuning-and-eval-tools
Please find this presentation on slideshare
https://www.slideshare.net/stefanosalsano/tuning-vim-performance-for-unikernels
37

38. The SUPERFLUIDITY project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement No.671566
(Research and Innovation Action).
The information given is the author’s view and does not necessarily represent the view
of the European Commission (EC). No liability is accepted for any use that may be
made of the information contained.
38