JULIAN BORREY
Coinbase is a company that empowers its developers to deploy fresh code to production just minutes after writing it yet there are has massive security requirements. Cryptocurrency companies are constantly being attacked, and Coinbase, which stores billions of dollars of irreversible cryptocurrency, is one of the biggest bounties on the internet. One of the pillars that allows us to maintain security in a CICD engineering organization is automated security scanning. Such scanners are often configured on a per-repository bases and may look for CVEs in dependencies or common anti-patterns that lead to vulnerabilities. In order for the Coinbase security team keep up with our ever growing product space, we built a tool that helps us centrally orchestrate our scanning pipeline on every project simultaneously. This tool is called Salus and is now being released free and open source.
It is not necessarily easy to integrate security scanners en masse. A security team will start by finding relevant scanners and then inserting them into a project’s test suite. At first, when Coinbase had just a few projects, custom configuration for each repository worked fine. Each time the security team wanted to use a new scanner, update scanner configuration or roll out new policies, we updated each repository. As Coinbase scaled and became more polyglot, the time it took to maintain our security scanners rose dramatically until it was untenable to maintain strong scanning on every repository. As David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Salus is our level of indirection to solve this problem. It is a docker container equipped with security scanners for many commonly used languages and frameworks as well a small ruby application used to coordinate the scanners. A developer can now add the Salus scanner to their test suite and on each build, it will pull down the latest Salus container, volume in their source code and execute the relevant scanners. We ensure that Salus results are immediately communicated to the developer and metrics about each project are communicated to the logging pipeline. Salus became a single place for the security team to make changes to the scanning pipeline that would be instantly applied org wide. Metrics aggregation also allowed for immediate insight into possible dangers as new vulnerabilities are discovered or to keep a pulse on the aggregate security posture of the company. Today, Ruby, Node, Python, Go, Shell and arbitrary pattern searches are represented in Salus and this will expand in the future as the project evolves. This talk aims to explain how an engineering team can start using Salus to enable them to stay safe with as little friction and effort as possible.
8. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
9. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works
10. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts
11. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts
● Salus details
○ Configuration
○ Custom scanners
○ Metrics & reporting
12. LONDON 18-19 OCT 2018
Overview
● Review of security scanners
● Problems with security scanners at scale
● What Salus does and how Salus works → free, no vendors/accounts
● Salus details
○ Configuration
○ Custom scanners
○ Metrics & reporting
● Pointers to source code & more resources
14. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
15. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Source Code
16. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
ScannerSource Code
17. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Scanner
Pass
Fail
Source Code
18. LONDON 18-19 OCT 2018
Software Scanners
A scanner is software that analyses other software.
Scanner
Pass
Fail
Source Code
19. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
20. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
21. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)
22. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)
23. LONDON 18-19 OCT 2018
Software Scanners
Linter - checks syntax follows certain policies.
Rubocop
(Ruby Linter)
24. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
25. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
CVE = Common Vulnerability Enumeration -
some documented vulnerability
26. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
27. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
28. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit
29. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit
30. LONDON 18-19 OCT 2018
Security Scanners
CVE scanner - looks for known vulnerabilities in dependencies
BundlerAudit
32. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
33. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
Verizon Data Breach Investigations Report, 2015
34. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
Verizon Data Breach Investigations Report, 2018
Verizon Data Breach Investigations Report, 2015
35. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
36. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans
37. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans
● Can run on every build round the clock
38. LONDON 18-19 OCT 2018
Security Scanners
Scanners are important:
● Powerful - can search huge CVE databases
● Some anti-patterns are obvious and scanners can do
the job. E.g. using `eval()` on user controlled input.
● Not fatigued like humans
● Can run on every build round the clock
● Not silver bullets, use in tandem with human review
40. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
$ cd /path/to/repo
$ bundle-audit check
41. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
$ cd /path/to/repo
$ bundle-audit check
Name: activesupport
Version: 3.2.10
Advisory: CVE-2013-1856
Criticality: High
URL: http://www.osvdb.org/show/osvdb/91451
Title: XML Parsing Vulnerability affecting JRuby users
Solution: upgrade to ~> 3.1.12, >= 3.2.13
42. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Execute the scanner manually.
● Could be slightly better with pre-commit hook
$ cd /path/to/repo
$ bundle-audit check
Name: activesupport
Version: 3.2.10
Advisory: CVE-2013-1856
Criticality: High
URL: http://www.osvdb.org/show/osvdb/91451
Title: XML Parsing Vulnerability affecting JRuby users
Solution: upgrade to ~> 3.1.12, >= 3.2.13
43. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
44. LONDON 18-19 OCT 2018
How might you deploy a scanner?
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
45. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer
46. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control
47. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control CI servers
48. LONDON 18-19 OCT 2018
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
How might you deploy a scanner?
developer source control CI servers
AWS / GCP
/ etc
production servers
49. LONDON 18-19 OCT 2018
How might you deploy a scanner?
...
“100% of services deployed to production must have a security scan.”
51. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
52. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.
53. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.
● So do it yourself?
54. LONDON 18-19 OCT 2018
Upgrading the fleet
● Have to make M x N code changes.
● Want to avoid asking the service owners:
○ Lots of work to keep asking for this.
○ Requires a fair bit of context to understand
the tool and configure it correctly.
● So do it yourself?
55. LONDON 18-19 OCT 2018
"All problems in computer science can be solved
by another level of indirection." - David Wheeler
56. LONDON 18-19 OCT 2018
Enter Salus
"All problems in computer science can be solved
by another level of indirection." - David Wheeler
57. LONDON 18-19 OCT 2018
How might you deploy a scanner?
Run Latest
Salus
Container
● Run the scanner in the CI/CD pipeline via the repo’s test suite.
59. LONDON 18-19 OCT 2018
What is Salus
$ docker run --rm -t -v $(pwd):/home/repo coinbase/salus
60. LONDON 18-19 OCT 2018
What is Salus
$ docker run --rm -t -v $(pwd):/home/repo coinbase/salus
● docker run
● --rm
● -t
● -v $(pwd):/home/repo
● coinbase/salus
63. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
How Salus works
64. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
How Salus works
65. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
● Compiles report, prints to STDOUT and HTTP post
How Salus works
66. LONDON 18-19 OCT 2018
Ruby app in container:
● Initializes with configuration (more on this later)
● Loops through each scanner
○ Ruby app? → run `bundle-audit check`
○ Rails app? → run `brakeman`
○ Node app? → run `npm audit`
● Compiles report, prints to STDOUT and HTTP post
● Exits !0 if issues are found (which could fail CI)
How Salus works
67. LONDON 18-19 OCT 2018
Why is Salus useful?
● Have to make M x N code changes.
68. LONDON 18-19 OCT 2018
Why is Salus useful?
● Have to make M x N code changes.
● Make 1 code change.
69. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
70. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
74. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
75. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)
76. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)
● Each scanner could also be customized
77. LONDON 18-19 OCT 2018
Salus Configuration
● Salus has a bunch of scanners:
○ BundlerAudit
○ Brakeman
○ npm audit
○ PatternSearch (grep)
● You can choose what fails a Salus run → leads to !0 exit status
(useful for CI pipelines to fail)
● Each scanner could also be customized
● Salus has a --config flag
78. LONDON 18-19 OCT 2018
Salus Configuration
● Config can also be provided via:
○ A salus.yaml file in the repository’s root will be automatically parsed.
○ URI in the environment variable SALUS_CONFIGURATION
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config file://tests/salus.yaml
80. LONDON 18-19 OCT 2018
Salus Configuration
● For global security policies that every repository should follow, use a remote URI.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml
81. LONDON 18-19 OCT 2018
Salus Configuration
● For global security policies that every repository should follow, use a remote URI.
● Especially useful for testing out new security policies before enforcing them.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml
82. LONDON 18-19 OCT 2018
Salus Configuration
● But what if we need to allow an exception to the global policy for just one repo?
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config https://internal.net/salus.yaml
83. LONDON 18-19 OCT 2018
Salus Configuration
● But what if we need to allow an exception to the global policy for just one repo?
● You can concatenate configuration files to allow for local customization.
docker run
--rm
-v $(pwd):/home/repo
coinbase/salus --config “https://internal.net/salus.yaml
file://tests/salus.yaml”
90. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
91. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
92. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)
93. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)
94. LONDON 18-19 OCT 2018
Salus Reports
Reports include:
● Which scanners pass/failed
● Reasons for failure
● Which dependencies are present
(name + verison + source)
● Which Salus configuration they it used
STDOUT (default)
95. LONDON 18-19 OCT 2018
Salus Reports
TXT format to STDOUT (developer) JSON format for consumer
96. LONDON 18-19 OCT 2018
Salus Reports
Screenshot of Kibana displaying the results of Salus scans
97. LONDON 18-19 OCT 2018
More resources
● Github: coinbase/salus
● Docker Hub: coinbase/salus
● Blog post: https://blog.coinbase.com/engineering/home
98. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
99. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
Security team can keep up
metrics
100. LONDON 18-19 OCT 2018
Why is Salus useful?
Latest Salus
Pass
Fail
Source Code
Container Registry or
Configuration Host
Quick developer
feedback loop
Security team can keep up
metrics
101. LONDON 18-19 OCT 2018
Thank you:
● Developers of open source scanners
● Ryan Sears, Adam Richardson, Slava
Kim - all contributors of Salus
● DevSecCon Organizers