As presented at Open Source Open Standards (GovNet) (http://opensourceconference.co.uk/), this deck covers some of the material which operators of open source data centers and users of container and cloud technologies should be aware of when seeking to be security conscious.
Traditionally, when datacentre operators talk about application security, there has been a tendency to focus on issues related to key management, firewalls and data access. By contrast, application developers have a security focus which is more aligned with code analysis and fuzzing techniques. The reality is, secure application deployment principles extend from the infrastructure layer through the application and include how the application is deployed. With the prevalence of continuous deployment, it’s imperative to focus efforts on what attackers’ view as vulnerable; particularly in an environment where new exploits are being disclosed almost daily.
In this session we’ll present:
- How known vulnerabilities can make their way into production deployments
- How vulnerability impact is maximized
- A methodology for ensuring deployment of vulnerable code can be minimized
- A methodology to minimize the potential for vulnerable code to be redistributed
2. #whoami – Tim Mackey
• Current roles: Senior Technical Evangelist; Occasional coder
• Previously XenServer Community Manager
• Cool things I’ve done
• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system
• Find me
• Twitter: @TimInTech ( https://twitter.com/TimInTech )
• SlideShare: slideshare.net/TimMackey
• LinkedIn: www.linkedin.com/in/mackeytim
6. EASY ACCESS TO SOURCE CODE
Open source ubiquity makes it ready target
OPEN SOURCE ISN’T
MORE OR LESS
SECURE THAN
CLOSED SOURCE –
ITS JUST EASIER TO
ACCESS
VULNERABILITIES ARE PUBLICIZED
EXPLOITS ARE PUBLISHED
7. Anatomy of a new attack
Potential Attack
Iterate
Test against platforms
Document
Don’t forget PR department
Deploy
8. DEVELOPER DOWNLOADS
OUTSOURCED DEVELOPMENT
THIRD PARTY LIBRARIES
CODE REUSE
APPROVED COMPONENTS
COMMERCIAL APPS
OPEN SOURCE CODE
Open source enters through many channels…
…and vulnerabilities can come with it.
9. CLOSED SOURCE COMMERCIAL CODE
• DEDICATED SECURITY RESEARCHERS
• ALERTING AND NOTIFICATION INFRASTRUCTURE
• REGULAR PATCH UPDATES
• DEDICATED SUPPORT TEAM WITH SLA
OPEN SOURCE CODE
• “COMMUNITY”-BASED CODE ANALYSIS
• MONITOR NEWSFEEDS YOURSELF
• NO STANDARD PATCHING MECHANISM
• ULTIMATELY, YOU ARE RESPONSIBLE
Who is responsible for code and security?
10. 0
500
1000
1500
2000
2500
3000
3500
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Open Source Vulnerabilities Reported Per Year
BDS-exclusive nvd
Reference: Black Duck Software Knowledgebase, NVD
Increasing number of OSS vulnerabilities
11. Automated tools miss most open source vulnerabilities
Static & Dynamic Analysis
Only discover common vulnerabilities
3,000+ disclosed in 2014
Less than 1% found by automated tools
Undiscovered vulnerabilities are
too complex and nuanced
All possible security
vulnerabilities
12. What do these all have in common?
Heartbleed Shellshock GhostFreak Venom
Since:
Discovered:
2011
2014
1989
2014
1990’s
2015
2000
2015
2004
2015
Discovered by:
Component: OpenSSL
Riku, Antti,
Matti, Mehta
Bash
Chazelas
OpenSSL
Beurdouche
GNU C library
Qualys researchers
QEMU
Geffner
16. Distinct areas of risk
• Open source license compliance
• Ensure project dependencies are understood
• Use of vulnerable open source components
• Is component a fork or dependency?
• How is component linked?
• Operational risk
• Can you differentiate between “stable” and “dead”?
• Is there a significant change set in your future?
• API versioning
• Security response process for project
17. Total Quality Management Philosophies
• Detect problems before product ships
• Select components based on trust
• Continuously identify issues and improve
• Empower employees to solve problems
• Implement the Deming Cycle
• Plan for change and analyze risk
• Do execute the plan in small steps
• Check the results against the plan
• Act on results to improve future outcomes
• Manage with facts
19. Software development lifecycle – threat model
Idea
Spec
Design
Code
Test
Release
• As part of the specification and
design, threat models are often
created.
20. Software development lifecycle – static analysis
Idea
Spec
Design
Code
Test
Release
• As part of the specification and
design, threat models are often
created.
• During code creation and
commits, static analysis is
performed
21. Software development lifecycle – dynamic analysis
Idea
Spec
Design
Code
Test
Release
• As part of the specification and
design, threat models are often
created.
• During code creation and
commits, static analysis is
performed
• Testing usually includes some
form of dynamic testing
23. Oops – a vulnerability is disclosed – now what?
DEVELOP SCM BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
REMEDIATE AND TRACK
LICENSE COMPLIANCE AND
SECURITY VULNERABILITIES
FULL APP SEC VISIBILITY
INTEGRATION
BUILD / CI SERVER
SCAN APPLICATIONS
WITH EACH BUILD VIA CI
INTEGRATION
DELIVERY PIPELINE
SCAN APPLICATIONS
AND CONTAINERS
BEFORE DELIVERY
CONTINUOUS
MONITORING OF
VULNERABILITIES
26. Knowledge is key. Can you keep up?
glibc
Bug
Reported
July 2015
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
27. Knowledge is key. Can you keep up?
glibc
Bug
Reported
July 2015
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
28. Knowledge is key. Can you keep up?
glibc
Vuln
Introduced
May 2008
glibc
Bug
Reported
July 2015
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
29. Knowledge is key. Can you keep up?
glibc
Vuln
Introduced
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
glibc
Bug
Reported
July 2015
National
Vulnerability
Database
Vuln
Published
Feb 18-2016
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
30. Knowledge is key. Can you keep up?
glibc
Vuln
Introduced
National
Vulnerability
Database
Vuln
Published
You
Find It
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016 Feb 18-2016
glibc
Bug
Reported
July 2015
Patches
Available
You
Fix It
Highest Security Risk
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
31. Knowledge is key. Can you keep up?
glibc
Vuln
Introduced
National
Vulnerability
Database
Vuln
Published
You
Find It
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016 Feb 18-2016
glibc
Bug
Reported
July 2015
Patches
Available
You
Fix It
Highest Security Risk
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
33. A complete solution …
Choose Open
Source
Proactively choose
secure, supported
open source
SELECT
Inventory
Open Source
Map Existing
Vulnerabilities
Maintain accurate list of
open source
components throughout
the SDL
Identify vulnerabilities
during development
VERIFY
Track New
Vulnerabilities
Alert newly disclosed
vulnerabilities in
production
MONITORREMEDIATE
Fix
Vulnerabilities
Tell developers
how to remediate
34. OVER TWO HUNDRED THIRTY
E M P L O Y E E S
27USE BLACK DUCK
SOFTWARE
AWARD FOR
INNOVATION
GARTNER GROUP
“COOL VENDOR”
INNOVATIVE TECHNOLOGY
OF THE YEAR - SECURITY
7 YEARS IN A ROW FOR
SECURITY INNOVATION
RANKED #38 OUT OF 500
SECURITY COMPANIES
7 YEARS IN A ROW
6 of the top 8 mobile handset vendors
7 of the top 10 SOFTWARE COMPANIES (44% OF TOP 100)
24 COUNTRIES
6 of the top 10 banks
FORTUNE 100
Black Duck Created an Industry
35. 8,500
WEBSITES
350
BILLION LINES OF CODE
2,400
LICENSE TYPES
1.5
MILLION PROJECTS
76,000
VULNERABILITIES
Comprehensive KnowledgeBase
• Largest database of open source project
information in the world.
• Vulnerabilities coverage extended through
partnership with Risk Based Security.
• The KnowledgeBase is essential for
identifying and solving open source issues.
36. We need your help
Knowledge is power
• Know what’s running and why
• Define proactive vulnerability response process
• Don’t let technology hype cycle dictate security
Invest in defense in depth models
• Don’t rely on perimeter security to do heavy lifting
• Do look at hypervisor & container trends in security
• Make developers and ops teams part of the solution
• Do embed security into deployment process
Together we can build a more secure data center
37. Free tools to help
• Docker Container Security Scanner
• https://info.blackducksoftware.com/Security-Scan.html
• 14 Day Free Trial to Black Duck Hub
• https://info.blackducksoftware.com/Demo.html
• Red Hat Atomic Host Integration (Requires Black Duck Hub)
• atomic scan --scanner blackduck [container]
Notas del editor
Image: http://morguefile.com/p/209940
http://www.istockphoto.com/photo/computer-crime-concept-gm516607038-89059287?st=9174601
Source: http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/
Every year since 2008, Verizon have published a report on the attempted data breaches occurring within their data centers. For 2015, they found close to 90% of them had either a financial or espionage component to them. This report is well worth the read, and there are a few key findings in this report we should all be aware of.
Costs of data breaches are heavily skewed towards legal consultation and forensics, and not to the public components of credit monitoring or lawsuits
Despite some vulnerabilities having been public for years, there remain vulnerable components in use
Some of those components simply may not have a patch forthcoming for a variety of reasons.
Despite years of organizations spending energy protecting against attacks, it remains up to the attacker to define what’s valuable. Consider the case of ransomware. A police department in the town next to where I live was subjected to a raonsomeware attack. For roughly 500 USD in bitcoin, the attackers would decrypt the booking and evidence records they had just crypto locked. As an attacker, they likely had no knowledge of who they had attacked or what they had locked up. What mattered was the ransom, and that they had a police organization’s files didn’t factor into the equation.
Let’s take a little bit of time and look at how an attack is created. Potential attackers have a number of tools at their disposal, and use a number of different tactics. In this case, the attacker wishes to create an attack on a given component. In order to be effective, they have two primary models. First they can actively contribute code in a highly active area of the component with an objective of planting a back door of some form. The hope being that their code will fail to be recognized as suspect given how quickly the area of code is evolving.
Second they can look for areas of code which are stable, and the longer they’ve bene stable, the better. The reason for this is simple, old code is likely written by someone who isn’t with the project any longer, or perhaps doesn’t recall all assumptions present at the time the code was written. After all, its been long understood that even with the best developers, assumptions change and old code doesn’t keep up.
The goal in both cases being to create an attack against the component, so they test, and fail, and iterate against the component until they’re successful or move on. Assuming they’re successful, they create a deployment tool and document the tool for others. Of course, given the publicity received by some recent vulnerabilities, a little PR goes a long way.
Now there are responsible researchers who follow a similar workflow, and they legitimately attempt to work with component creators to disclose vulnerabilities. They too will publish results, but are less interested in creating the an attack beyond a proof of concept.
http://www.istockphoto.com/photo/person-in-hooded-sweater-using-a-laptop-on-wooden-table-gm464503138-58544934?st=cf78f31
http://www.istockphoto.com/photo/cloud-computing-gm518556682-90104967
https://www.cesg.gov.uk/guidance/open-source-software-%E2%80%93-exploring-risk-good-practice-guide-38
If you’re using commercial software, the vendor is responsible for best practice deployment guidance, the notification of any security vulnerabilities and ultimately patches and workarounds for disclosed vulnerabilities. This is part of the deliverable they provide in return for their license fee.
If you’re using open source software, that process becomes partly your responsibility. To illustrate the level of information you have to work with, let’s look at a media-wiki maintenance release from December 2015.
“various special pages resulted in fata errors” – this clearly is something which needs resolution, but which pages? How do you test?
“1.24.6 marks the end of support for 1.24.x” – this is good to know, but I hope it was published elsewhere.
“However, 1.24.5 had issues (along with other versions) so it was thought fair to fix them” – This is a good thing, but can we expect this treatment in the future? From the title, we also have a fix for 1.23.x, but what other versions?
There is one thing we should all notice from this data: The vulnerable code was present for years until discovery. What may not be known, is that these vulnerabilities were found by researchers, not analysis tools.
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
On July 13 2015, the bug report associated with what would ultimately become CVE-2015-7547 was created.
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
The report indicated that a traditional buffer management issue was present. Specifically it said “this change causes the thisanssizp pointer variable used in the recvfrom function on line 1282 to use the wrong size if a new buffer is created after the thisanssizp address has been changed at line 1257” and indicated that the result would be “The program will crash if the calculated size of the buffer used is 0. The recvfrom function will not crash, but any further accesses to the buffer where the bytes read was 0 from the recvfrom function will crash the program. ”
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
On Feb 16, 2016, a CVE assignment was made to the bug list, and it further indicated the problem was introduced in May of 2008 in version 2.9.
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
It wasn’t until two days later that the NVD (national vulnerability database) was updated to reflect the vulnerability. This meant that a staggered awareness situation ensued. When disclosures are staggered, there is an increased potential that a bad actor can take advantage of the situation. It’s important to note that this NVD entry came through US-CERT. One important thing to notice is that unlike the bug report, the overview contains significantly more actionable information (e.g. indicating that DNS is impacted “a crafted DNS response that triggers a call to the getaddrinfo function with the AF_UNSPEC or AF_INET6 address family, related to performing "dual A/AAAA DNS queries" and the libnss_dns.so.2 NSS module”
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
Of course all most data center operators really want is to secure their environment, so the question really becomes “how to do that”. For an indication of how that plays out, we’ll look at VMware’s response as an example. The first thing most vendors do is create some form of security advisory, which in this case is “VMware Knowledge Base article 2144032”. In it they list the then current information about their exposure to the vulnerability. Often times, it’s minimal, but as they investigate, details are fleshed out. On February 22nd, they amended the KB article and created a security advisory “VMware Security Advisory VMSA-2016-0002” which included details on a patch for ESXi 5.5 and then updated a day later for a patch in ESXi 6.0. It’s not the least bit uncommon for some subordinate products, including older versions, to take longer to patch and on March 29th updates were announced to cover “vCenter Server Appliance (VCSA), 5.0 U3f, 5.1 U3c, and 5.5 U3c”
Once you’ve a patch, then you can start working on resolving the issue in your infrastructure.
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://www.youtube.com/watch?v=hkryI6eapOA
http://blogs.vmware.com/security/2016/02/vmware-products-and-cve-2015-7547-glibc-getaddrinfo-security-issue.html
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
From a timeline perspective, we’re looking at close to eight years from time of bug to vulnerability resolution. For much of that time, the impact was unknown. Starting in July of 2015, the scope of the issue started to be known, but close to seven months were required for investigation, triage, development of a fix and public disclosure to occur. Once the bug was known, the risk of exploitation increased slightly until that disclosure, but following the disclosure risk goes up dramatically. That’s one reason why knowing what’s running in your environment is so important. The overall goal being to reduce the time between disclosure and fix.
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://www.youtube.com/watch?v=hkryI6eapOA
Source: Future of Open Source 2016 survey: https://www.blackducksoftware.com/2016-future-of-open-source
Docker Container Security Scanner
https://info.blackducksoftware.com/Security-Scan.html
14 Day Free Trial to Black Duck Hub
https://info.blackducksoftware.com/Demo.html
Red Hat Atomic Host Integration (Requires Black Duck Hub)
atomic scan --scanner blackduck [container]