Recent vulnerabilities like Heartbleed and Shellshock have brought the security practices and track record of open-source projects into the spotlight. A project’s response to security issues has a major impact on how much risk end users are exposed to and how the project is perceived in the technology industry. We will compare the security practices of key projects such as Linux, Docker, Xen Project, OpenStack and others. We will explore the trade-offs of different security practices, such as community trust, competing stakeholder interests, fairness and media coverage of vulnerabilities. Finally, we will explore the evolution of the Xen Project’s security process over the past 3 years as a case study. We will illustrate the trade-offs, pain points and unexpected issues we have experienced, to help other projects understand the pit-falls in designing robust security processes and help users of open source projects understand how open source projects manage security vulnerabilities.

1. Lars Kurth
Community Manger, Xen Project
Chairman, Xen Project Advisory Board
Director, Open Source/Xen Project, Citrix
lars_kurth

2. Was a contributor to various projects
Worked in parallel computing, tools,
mobile and now virtualization
Community guy for the Xen Project
Working for Citrix
Member of the group that develops XenServer
Chairman of Xen Project Advisory Board

4. I
I: Vulnerability Introduced
D: Vulnerability Discovered
D
Discoverer exploits
issue for his own
purpose

5. I
I: Vulnerability Introduced
D: Vulnerability Discovered
Discoverer reports security issues to
security@yourproject
D

6. A team-effort to ensure that …
• All (known) doors are closed
• All (known) doors are locked
• All (known) windows are
boarded up
• Fences have no (known)
weaknesses
• …

7. XF
R: Vulnerability Reported
T: Triage
A: Vulnerability Announced
F: Fix Available
X: Fix Deployed
Vulnerability is known by the reporter and the security team
Note: It may also be known and used by black hats
Vulnerability is known publicly with no fix available
Vulnerability is known publicly with fix available
Basic
Description
R T A
Patch/fix creation
and validation

8. X
R: Vulnerability Reported
T: Triage
P: Vulnerability Pre-disclosed
A: Vulnerability Announced
F: Fix Available
X: Fix Deployed
Vulnerability is known by the reporter and the security team
Note: It may also be known and used by black hats
Vulnerability is known about by a privileged and small group of users
Vulnerability is known publicly
A
Pre-disclosure period
R P
Patch/fix creation
and validation
FT

9. Users Safe

10. Encourage discoverers to report security issues
to security@yourproject
Discoverers are in control
You can’t stop them from releasing/using information
A robust vulnerability process encourages discoverers to work with you

11. Ensure that your project fixes security issues
as quickly as possible
You don’t want unaddressed vulnerabilities

12. Exposure time to security issues is minimized
A maximum of users* apply patches quickly
Minimize risk

13. Linux Kernel/LXC/KVM if reported via OSS Security
Linux Kernel/LXC/KVM if reported via security@kernel.org
OpenStack, QEMU, … for low impact issues
Full
Linux Kernel/LXC/KVM if reported via OSS Security Distros
Linux Distributions (both open source and commercial)
QEMU, Libvirt, oVirt, ...
OpenStack for intermediate to high impact issues
OPNFV, OpenDayLight : process modeled on OpenStack
Xen Project for all issues (also handles 3rd party issues, e.g. QEMU)
Docker : states responsible disclosure; but policy docs empty / some CVEs
Responsible
Cloud Foundry : no clearly stated process; no published CVE’s
CoreOS: just a mail to report issues
Kubernetes: : just a mail to report issues (when I wrote this talk in Aug, no info)
Not clearly
stated
Approach Used by Projects

14. Open-source software projects are often well
intended, but security can take a back seat to
making the code work. OpenDaylight, the
multivendor software-defined networking
(SDN) project, learned that the hard way last
August after a critical vulnerability was found
in its platform. It took until December for the
flaw, called Netdump, to get patched …
PC World, March 2015

15. Using the pre-dominant model as baseline
Applies to Linux Distros, OSS Sec Distros, QEMU, …
Mike Licht @ Flickr

16. A X
Typically fixed time during which the security issue is handled secretly
Depends on discoverer’s wishes
R: Vulnerability Reported
T: Triage
P: Vulnerability Pre-disclosed
A: Vulnerability Announced
F: Fix Available
X: Fix Deployed
Vulnerability is known by the reporter and the security team
Note: It may also be known and used by black hats
Vulnerability is known about by a privileged and small group of users
Vulnerability is known publicly
Description, CVE
allocation, …
Pre-disclosure period
R
Patch/fix creation
and validation
FT P
What can and can’t be done with
privileged information can differ
significantly between projects

17. mindfulness @ Flickr

18. F A XR
Disclosure Time

19. Long disclosure times discredit responsible disclosure
From a few days to many months
Long disclosure times create a disincentive for reporters to work with you
Increases the risk of 0 day exploits
Pre-defined disclosure times help manage vendors
Example later
Most successful projects have a 2-3 weeks disclosure period

20. Assigning CVE numbers is best practice in by
established projects and vendors in the
Linux/Cloud ecosystem

21. CVE databases (such as www.cvedetails.com) can be used
to evaluate your project
This shows Xen Project CVE stats
Before 2012, we didn’t have fewer vulnerabilities than after
We just didn’t have a process requiring creation of CVEs

22. A fair comparison between projects/technologies using CVE
data is not easily possible
Not all projects/products create CVEs for all their issues
Example: Linux/QEMU only do so for severe ones
Policies are not always published
Some projects don’t assign CVEs at all
Some technologies/products cannot be easily identified in databases
Example: KVM, LXC
Sometimes CVEs can affect several products
But are counted only against one
Open source product definitions on cvedetails are often sloppy

23. Mike Licht @ Flickr

24. Description, CVE
allocation, …
A D
Pre-disclosure period
R
Patch/fix creation
and validation
FT P
What happens here depends
on your process goals

25. Make sure that a fix is available before disclosure
Make sure that downstream projects and products (e.g. distros) can
package and test the fix in their environment
Allow service providers that use your Software to start planning an
upgrade (at scale this can take a week)
Allow service providers that use your Software to deploy an upgrade
before the embargo completes

26. What is allowed during pre-disclosure
Who is privileged and trusted to be on the pre-disclosure
mailing list
Disclosure Time

27. Make sure that a fix is available before disclosure
Make sure that downstream projects and products (e.g. distros) can
package and test the fix in their environment
Allow service providers that use your Software to start planning an
upgrade (at scale this can take a week)
Allow service providers that use your Software to deploy an upgrade
before the embargo completesCloud Model
Distro Model

28. Emerged recently!
Recognizes the needs of service providers
Pre-Cloud Computing!
Services and their users are vulnerable
immediately after disclosure

29. Approach Used by Projects
Linux Kernel/LXC/KVM if reported via OSS Security Distros
Linux Distributions (both open source and commercial)
QEMU, Libvirt, oVirt, ...
OpenStack for intermediate to high impact issues
OPNFV, OpenDayLight : process modeled on OpenStack
Xen Project for all issues (also handles 3rd party issues, e.g. QEMU)
Docker: depends on severity, details only available on request

31. More Cloud/Service users than direct users of your software
Example:
AWS stated in 2014 that they have > 1M users (and a lot more instances)
AliCloud claims that they have > 1M users
…

32. Just imagine what the reputation damage would have been, if Xen had put AWS,
Rackspace, SoftLayer, … users at real risk of a vulnerability.
There were 100’s of
stories at the time,
despite the fact that
users were never put
at risk, but merely
inconvenienced !

33. Pre-disclosure list membership:
more members, more risk of leakage
In the Distro Model, the number of privileged users is typically <10
In the Cloud Model, the number could be an order of magnitude higher (50-100)
This increases risk of information being accidentally released

34. Restricting pre-disclosure list membership
Restricting membership to large service providers to minimize risk
That creates issues of “fairness”
Which may be incompatible with your communities' values

35. How the Xen Project got to its
Vulnerability Process
xenproject.org/security-policy.html
Moyan Brenn @ Flickr

36. 2011 2012 2013 2014 2015 2016
Goals:
Allow fixing, packaging and testing;
Allow service providers to prepare (but not deploy) during embargo
Pre-disclosure:
Membership biased towards distros & large service providers
No predefined disclosure time
1.0

37. 2011 2012 2013 2014 2015 2016
July 2012: CVE-2012-0217, Intel SYSRET
Affected FreeBSD, NetBSD, Solaris, Xen and Microsoft Windows
A large pre-disclosure list member put pressure on
key members of the Xen Project Community to get an embargo
extension
They eventually convinced the discoverer to request an extension
1.0

38. 2011 2012 2013 2014 2015 2016
Centered on:
Predetermined disclosure schedule: 1 week to fix, 2 weeks embargo
Who should be allowed on the pre-disclosure list
Fairness issues between small and large service providers
Direct vs. indirect Xen consumers
The risk of larger pre-disclosure list membership
1.0

39. 2011 2012 2013 2014 2015 2016
Strongly recommended disclosure schedule
Inclusive pre-disclosure list membership
Changes to application procedure (based on checkable criteria)
1.0 2.0

40. 2011 2012 2013 2014 2015 2016
Sept 2014: CVE-2014-7118
Leading to the first Cloud Reboot
AWS pre-announced cloud reboot to their customers
Other vendors didn’t.
Policy was interpreted differently by vendors.
This highlighted ambiguities in the project’s security policy
(what can/can’t be said/done during an embargo)
1.0 2.0

41. 2011 2012 2013 2014 2015 2016
Goals:
Allow fixing, packaging and testing
Allow service providers to prepare (and normally to deploy) during embargo
Pre-disclosure:
Clearer application criteria
Public application process (transparency)
Clear information on what is/is not allowed during an embargo (per XSA)
Means for pre-disclosure list members to collaborate
1.0 2.0 3.0

42. 2011 2012 2013 2014 2015 2016
Conducted XSA-133 Retrospective upon request
Process change: Earlier embargoed pre-disclosure without patches
May 2015: CVE-2015-3456
First time we were affected by a branded bug
QEMU bug, which was handled by several security teams: QEMU,
OSS Distro Security, Oracle Security & Xen Project
From a process perspective: were not able to provide a
fix 2 weeks before the embargo date ended
1.0 2.0 3.0

43. Larger pre-disclosure list has not caused a single issues in two years of
operating an inclusive approach
We have not had a single 0-day vulnerability
A well run vulnerability process builds trust
Willingness to adapt to your stake-holders needs builds more trust
It creates collaboration and understanding of stake-holders
Fairness is a difficult issue
There will always be practical issues, e.g. “interpretations of policy”, etc.

44. The Xen Project’s process is the only example case, where this issue
has been tackled through a community consultation.
To Contrast:
OpenStack does not publish who is on their pre-disclosure list
OpenStack does not have a formal application process
Avoids dealing with the “fairness” issue head-on

47. Security stories are “hot”
Xen is widely used, thus security stories “sell”
It’s too easy for reporters to write a story
Reporters just have to check our page,
and know when the next story comes

48. Source: yanilavigne.net via
domics.me

49. Very wide range of approaches vs.
The reality that SW stacks contain many layers
Consider the weakest link in your SW stack
Best Practice appears to be emerging
Older projects seem slow to change
New projects, don’t build security management into their culture from the
beginning
New Post-Snowden era pressures
How to effectively deal with media Hype?