Vulnerability Management Nirvana: A Study in Predicting Exploitability When everything is a priority, nothing is. 15% or 10,000 vulnerabilities have a CVSS score of 10. Vendors and practitioners alike use CVSS or their own threat intelligence models to predict which vulnerabilities will be exploited next. We review current options, present a predictive data-driven prioritization model, and how attendees can get started using our approach in their vulnerability management program.

1. Vulnerability
Management Nirvana
A STUDY IN PREDICTING EXPLOITABILITY
Michael Roytman
Senior Data Scientist
Kenna Security
MRoytman
Kymberlee Price
Senior Director of Operations
Bugcrowd
Kym_Possible
David F. Severski
Mgr., Information Security Program
Seattle Children’s Hospital
DSeverski

2. Story time!
a/k/a “Michael’s not here to stop David from putting in cat photos”
2

3. Let’s (re-)experience Stagefright!
CVE-2015-3867
3

4. On what day was Stagefright (CVE-2015-3867) disclosed?
4
August 15, 2015

5. Why Prioritization Matters
You have 150 vulnerabilities open with CVSS 6.8 and
above
Your inbound new vulnerabilities average 15 dev tasks
per week, from both internal and external sources
What do you fix first?
5

6. Historical Research: Prioritizing Product
Vulnerabilities
Sources of vulnerabilities
◦ Internal Security Research Group
◦ External Security Researchers
◦ Third Party Libraries/OSS Disclosures
Developers would prioritize on CVSS v2 Base Score
Limitations in CVSS became apparent
6

7. Historical Research: Prioritizing Product
Vulnerabilities
Applied custom criteria for extended CVSS fields
Weighted extended CVSS fields to adjust base CVSS
Defined priority bands with SLA for remediation
Automated the priority calculations – the only manual requirement was for the CVSS score to be
entered when the bug was logged, which was part of existing SOPs
7

8. Our Path towards Nirvana
Using CVSS for prioritization today
Alternative prioritization models
Our research
Comparative data & results
Conclusions & How to Apply
8

9. CURRENT VULNERABILITY
PRIORITIZATION MODELS

10. CVSSv2: The Tool We Have
Open industry standard
◦ Maintained and regularly updated
Modular – base, temporal, and environmental components
Objective… mostly
False negatives
Base score is non-predictive single point in time measure
Few companies use update Temporal or Environmental scores
No indication of historical attack patterns
Misused as prioritization tool, designed to convey vulnerability characteristics
10
The Common Vulnerability Scoring
System (CVSS) provides an open
framework for communicating the
characteristics and impacts of IT
vulnerabilities.

11. CVSSv3: The Tool We Need?
A few wording changes: Authentication  Privileges Required, Access Complexity  Attack
Complexity
Adds new dimensions to measure User Interaction and Scope
11
Intuitively, one may think of a scope change as breaking out of a sandbox, and an example would be a
vulnerability in a virtual machine that enables an attacker to delete files on the host OS (perhaps even its own
VM). In this example, there are two separate authorization authorities: one that defines and enforces
privileges for the virtual machine and its users, and one that defines and enforces privileges for the host
system within which the virtual machine runs.
A scope change would not occur, for example, with a vulnerability in Microsoft Word that allows an attacker to
compromise all system files of the host OS, because the same authority enforces privileges of the user's
instance of Word, and the host's system files.

12. CVSSv2 vs CVSSv3: CVE-2016-0800
(DROWN)
CVSS Severity (version 3.0):
CVSS v3 Base Score: 5.9 Medium
Vector: AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
Impact Score: 3.6
Exploitability Score: 2.2
CVSS Version 3 Metrics:
Attack Vector (AV): Network
Attack Complexity (AC): High
Privileges Required (PR): None
User Interaction (UI): None
Scope (S): Unchanged
Confidentiality (C): High
Integrity (I): None
Availability (A): None
12
CVSS Severity (version 2.0):
CVSS v2 Base Score: 4.3 MEDIUM
Vector: (AV:N/AC:M/Au:N/C:P/I:N/A:N)
Impact Subscore: 2.9
Exploitability Subscore: 8.6
CVSS Version 2 Metrics:
Access Vector: Network exploitable
Access Complexity: Medium
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of
information

13. CVSSv3: The Tool We Need?
Still a non-predictive single point in time measure
Temporal & Environmental fields now impact Base Score
◦ Base score is worst possible outcome – completing temporal and environmental fields only has potential
to lower base score
Framework transition pain
◦ Retooling your systems for new format
◦ No standardization to compare v2 vulns to v3 vulns
Still designed for communication of vulnerability characteristics
Relevance for IoT questioned1
13

14. Exploit Index: Is This Nirvana?
Exploitability Index (Microsoft, 2008)
◦ Intended to provide customers with more granularity to improve risk assessment and patch
prioritization
◦ Determining exploitability is heavily dependent on human researchers, creating scale and skill
limitations
July 2013, 5 years after Exploit Index launch:
“While no exploit surfaced for a vulnerability within 30 days of security
bulletin release, it does not mean that the vulnerability could not have
been exploited researchers or attackers may just have been prioritizing
other vulnerabilities instead”3
14

15. What Else Is There?
Indicators of Badness
◦ Exploit Presence in Metasploit
◦ Exploit Presence in Canvas
◦ Known Public vs Private Exploits
◦ Attack Vectors
15
That’s a lot of threat intelligence feeds to monitor and investigate in
real time on every vulnerability you’ve got logged.

16. NIRVANA RESEARCH

17. 50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations
Predicting Exploitability – Big Data Time
17

18. Predicting Exploitability – Big Data Time
150,000,000 Breaches
18

19. Baseline ALLTHETHINGS!!
Probability
(You Will Be Breached On A Particular Open Vulnerability)?
𝑂𝑝𝑒𝑛 𝑉𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑖𝑒𝑠 𝐵𝑟𝑒𝑎𝑐ℎ𝑒𝑠 𝑂𝑐𝑐𝑢𝑟𝑒𝑑 𝑜𝑛 𝑇ℎ𝑜𝑠𝑒 𝐶𝑉𝐸)
𝑇𝑜𝑡𝑎𝑙 𝑂𝑝𝑒𝑛 𝑉𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑖𝑒𝑠
19
6%

20. Probability a Vulnerability Having CVSS >
X Has Observed Breaches
0 2 4 6 8 10 12
10
9
8
7
6
5
4
3
2
1
0
Breach Probability (%)
CVSSBase
20

21. Probability a Vulnerability Having
Property X Has Observed Breaches
0 5 10 15 20 25 30 35 40
EDB+MSP
MSP
EDB
CVSS 10
Breach Probability (%)
21

22. VulnPryer: A Survival Strategy for Vulnerability
Management
Triaging and answering the question – “What should enterprise defenders fix first?”
22

23. Design Requirements
Use commonly accessible data
Customizable for our threat scenarios
Easy to produce
Must adjust to changing information
23

24. We Have the Data…We Can Rebuild It
National Vulnerability
Database
Network Security Posture
Analysis
Commercial Vulnerability
Feeds
Internal Asset Valuations
Sources of Data
24

25. VulnPryer Flow
CVSS Base
Normalize Base
Score
Metasploit factor Add EDB factor
Add Private Exploit
factor
Remove Network
factor
Remove Availability
and Integrity Only
factor
Adjusted CVSS Score
25

26. Automate ALLTHETHINGS!!
Daily Generation of
Vulnerability
Reference Library
•AWS Data Pipeline
•Risk Based Security VulnDB
•VulnPryer
Network Analysis
with Customized
Severities
• RedSeal
• Combine Asset Values with Actual
Network Configuration
Internal Reporting
and Prioritization
• Internal Dashboards
• Tableau and R for
Presentation
26
Fix
before
that

27. A Few Individual Results
Mean adjustment: -1.7 (24% decrease)
Maximum increase: 3.3 (112% increase)
Maximum decrease: 5.6 (99% decrease)
CVE-2014-0160 (Heartbleed) rescored from 5.0 to 8.3
4% of vulnerabilities reduced to CVSS 0
Approximately 18% reduction in network based risk scores
27

28. Results Over Our Specific Population
VULNPRYER-ADJUSTED CVSS BASE CVSS
28

29. Future Directions towards Nirvana
There’s always another layer…
Performance Optimization
Reporting
◦ Alerting on Changes to Scores
◦ Sample Reporting Templates (Tableau and/or R)
More Flexible Formula Changes
Generalize the Pythonic Framework for Other Use Cases
Analysis of Nirvana model in product security environment
29

30. CONCLUSIONS &
HOW TO APPLY

31. Conclusions
Base CVSS as sole criteria has serious known limitations
Readily available tools and data sources can be used to help you focus on what matters to you
and your organization
It is both possible and practical to stay abreast of changing vulnerability risk to drive timely
resource allocation decisions
31

32. Searching for Nirvana at Home
Easy to get started, just add
◦ Database of vulnerability features
◦ Your vulnerabilities
◦ A little bit of code
Code provided for you!
◦ VulnPryer in both Python and R versions
◦ Fully functional, automatic reference deployment via
Chef and AWS
32

33. QUESTIONS

34. REFERENCES

35. References – Tools and Code
VulnPryer
◦ Python Code – https://github.com/SCH-CISM/VulnPryer
◦ R Version – https://github.com/SCH-CISM/vulnpryr
◦ AWS Automation Code – https://github.com/SCH-CISM/sch-vulnpryer-orchestration
Vendors
◦ RedSeal – Network Security Posture Analysis
◦ Risk Based Security – Vulnerability Database
◦ Kenna Security – Prioritization as a service
35

36. References – Additional Reading
1 CERT/CC Blog: CVSS and the Internet of Things https://insights.sei.cmu.edu/cert/2015/09/cvss-
and-the-internet-of-things.html
2 Immunity, Inc. White Paper: A Bounds Check on the Microsoft Exploitability Index
http://download.microsoft.com/download/3/E/B/3EBDB81C-DF2F-470B-8A64-
981DC8D9265C/A%20Bounds%20Check%20on%20the%20Microsoft%20Exploitability%20Index
%20-%20final.pdf
3 Exploitability/Priority Index Rating Systems (Approaches, Value, and Limitations)
https://www.riskbasedsecurity.com/reports/RBS-ExploitabilityRatings-2013.pdf
36