Jose Lausuch, Ericsson, Nikolas Hermanns, Ericsson How can we make sure that new code in OPNFV does not break or stop CI? How can we ensure quick feedback for each patch-set? With the new way to snapshot a virtual deployment it is now possible to get virtual clouds up and running in about 2 min. In addition, through low amount of disk/cpu consumption and isolation of the networking it is possible to have a very high number of virtual deployments co-existing in the same bare-metal server.

2. Testing, CI Gating & Community
Fast Feedback: The Challenge
of Integration Projects
Nikolas Hermanns
Jose Lausuch

3. Outline
• Challenges of the SDNVPN project
• XCI patchset verification
• Virtual testing setup
• Functest test challenge

4. Integration Project (e.g. SDNVPN)
ODL SDNVPN Fuel
Apex
• Consume many different
Project
• Different release dates
• Different Bug tracking
• Different source code handling
• Different programming language
• Etc.

5. Challenge of Integration Projects (e.g. ODL in OPNFV)
ODL-Master
time
ODL-Release
OPNFV-Integration
•Bug finding
ODL-Release
Bug1
ODL-Stable
Bug2
Bug1 Bug2
ODL-Master
x
• Bug 1 found and
solved
• Bug 2 found but not
valid for master
anymore
• Bug 2 was not worth
the effort
• Bug 3 will not be
found until next
release
ODL is consumed too late!
Bug3

6. Challenge of Integration Projects (e.g. ODL in OPNFV)
Developer A
Bug 1
time
ODL-Release
ODL-Release
Bug1
Integrator A
• Dev A expert in Part of
ODL
• Int A expert in Fuel but
doesn’t know much about
ODL code base
• Int A needs to find and
understand the bug!
• Int A need to talk to Dev A
and explain the found bug
• Might be Dev A does not
even remember

7. Solutions
• Solution 1: More test cases in consumed project
• Integrator needs to convince and explain the test to Developer
• Test case might be too E2Eish to be created in consumed project
• Solution 2: XCI patchset verification
• Automated integration (Consume latest state)
• Fast feedback (reducing from months to < 1 hour)
• No interaction from integrator

8. New
Patchset
(Commit)
Create Virtual
Testing Platform
Download/Install
new artifact
Use standard
testing framework
e.g. Functest
Build ODL
artifact
10 min 15 min 35 min
Unit test
Installer Project
e.g. Apex
Snapshot
5 min
Positive -> Merge commit
Negative -> Prevent merge
Investigate the
logs/Find bugs
ODL OPNFV
Feedback

9. Bare Metal (48 threads/256 GB RAM/500 GB SSD)
vJenkins
slave
vController vCompute
vCompute
vCompute
Thread pool – 8 vCPU
 10 GB Diskspace
 40 GB RAM
 Own OVS networks
vJenkins
slave
vController vCompute
vController vCompute
Snapshot
Controller
vController
vController
vController
Snapshot
Compute
vCompute
vCompute
vCompute
~ 100 GB Diskspace
x6
• RAM and CPU limiting factor
• Very low disk IO cause no
installation process
• ~2 mins until Openstack/ODL
services up
• Easy exchange of snapshot

10. Using the Snapshot
vController
vController
vController
Snapshot
Controller
vController
vController
vController
~ 100 GB Diskspace
>1GB disk consumption per vHost
• Using the base concept of qemu-img

11. CPU pinning
• 4
• 3 8 vCPUs ? -> YES! – sharing is the key.
• 1
time
vJenkins
slave
vController
vCompute
CPU-Load
Jenkins-
Slave
running
8 vCPU
Start
vCmp
vCtrl
Started
vCmp
vCtrl
Testing
started
Testing
more
API
Under-
provisioned

12. Cpu pinning
• Other vPod do not get effected

13. Networking
• OVS for complete
isolation
• vPods using same
IPs so every vPod
has its own
networks
• Jenkins-slave for
Gateway doing NAT
vJenkins
slave
vController vCompute
Bare metal
host
admin
storage
private
public
host
NAT
internet

14. Bare Metal (48 threads/256 GB RAM/500 GB SSD)
vJenkins
slave
vController vCompute
vCompute
vCompute
Thread pool – 8 vCPU
 10 GB Diskspace
 40 GB RAM
 Own OVS networks
vJenkins
slave
vController vCompute
vController vCompute
Snapshot
Controller
vController
vController
vController
Snapshot
Compute
vCompute
vCompute
vCompute
~ 100 GB Diskspace
x6
• RAM and CPU limiting factor
• Very low disk IO cause no
installation process
• ~2 mins until Openstack/ODL
services up
• Easy exchange of snapshot

15. Example of application: Functest
FUNCTEST
pep8
unit
tests
pylint
docs
py35
py27
test new
code on
OPNFV
deployment

16. Example of application: Functest
• Functest bug might break CI
• Needs a way to test each change before merging
• Fast feedback to the commit owner
• Easy to validate framework changes on Functest
• Can be tested on a basic nosdn-nofeature scenario

17. Example of application: Functest
Challenges:
• Changes on Feature tests  need specific scenario deployed
• Increase verification time of each patchset ( + time to build a
docker image)
• Identify where the change is (framework, test case, …) to trigger
an appropriate test case

18. Thank You
Nikolas Hermanns Nikolas.Hermanns@ericsson.com
Jose Lausuch jose.lausuch@ericsson.com