The SolarWinds attack brought additional scrutiny software supply chain security, but concerns about organizations’ software supply chains have been discussed for a number of years. Development organizations’ shift to DevOps or DevSecOps has pushed teams to adopt new technologies in the build pipeline – often hosted by 3rd parties. This has resulted in build pipelines that expose a complicated and often uncharted attack surface. In addition, modern products also incorporate code from a variety of contributors – ranging from in-house developers, 3rd party development contractors, as well as an array open source contributors. This talk looks at the challenge of developing secure build pipelines. This is done via the construction of a threat model for an example software build pipeline that walks through how the various systems and communications along the way can potentially be misused by malicious actors. Coverage of the major components of a build pipeline – source control, open source component management, software builds, automated testing, and packaging for distribution – is used to enumerate likely attack surface exposed via the build process and to highlight potential controls that can be put in place to harden the pipeline against attacks. The presentation is intended to be useful both for evaluating internal build processes as well as to support the evaluation of critical external vendors’ processes.

1. Threat Modeling the CI/CD
Pipeline to Improve Software
Supply Chain Security
Dan Cornell
CTO, Denim Group

2. Agenda
• Risk Exposure from Software Build Pipeline
• Constituents and Stakeholders
• Drivers for Change
• Threat Modeling Overview
• Supporting Infrastructure
• Pipeline Threat Model
• Using the Threat Model
• Questions

3. Notable Incidents
• SolarWinds
• Compromised server in build infrastructure
• CodeCov
• Compromised container server
• Potentially spawning Executive Order, other government activity

4. Fundamental Disconnect
• We have developed tremendous tooling and automation that allows
us to create secure, reliable software at a scale not previously
considered
• The way we deploy and deliver this tooling puts us in a situation
where we can't actually trust anything that comes out of it
• Oops

5. Fractal Attack Surface
The closer you look, the more you find

6. Confidentiality Impact
• IP disclosure
• Leaked secrets
• Vulnerability disclosure

7. Integrity Impact
• Backdoors
• Other unwanted behaviors
• Compromised cryptography

8. Availability Impact
• Development pipeline infrastructure is hyper-connected
• Release delays
• Absence or misbehavior of supporting services

9. Threat Modeling Overview
• Full overview beyond the scope of this talk
• Some background for reference
• https://www.denimgroup.com/resources/blog/2014/07/modeling-builders-breakers/
• Let's learn as we go

10. Supporting Infrastructure
• Fixed assets vs ephemeral assets
• Scanning/patching regimen
• Who is responsible for this infrastructure?
• "Corporate" or a DevOps team?

11. The Perimeter is the Problem
• Critical part of threat modeling process – what is in/what is out?
• For modern CI/CD pipelines: turns out to be maddeningly hard
(impossible) to determine the boundary
• Many steps with lots of moving parts
• Dynamic attack surface: tomorrow may be different than today
• DevOps team members can often add stages/tools/elements with minimal supervision
• Fractal complexity
• Approach: Start with code commit and follow the workflow

12. Example CI/CD Pipeline Dataflow

13. General/Overarching Concerns
• Lots of network traffic
• Protect with TLS to guard Confidentiality, Integrity
• Lots of authentication points / IAM concerns
• Often multiple methods/avenues per system: interactive + API (+ other?)
• User <-> System
• System <-> System
• Is there a comprehensive IAM paradigm in place, or a combination of
overlapping approaches?

14. General/Overarching Concerns
• Data storage concerns
• How is data at rest protected? (Especially file/block storage)
• What are your unknown-unknowns
• External services and provides that are (generally) unknown but part of the
process
• Any developer with a GitHub account is now their own purchasing agent

15. Source Repository and Workflow Engine

16. Source Repository and Workflow Engine
• Sources of (custom) code
• In-house developers
• 3rd party development teams
• Likely different risk profiles
• Do contributions go through different workflows?

17. Source Repository and Workflow Engine
• Authentication
• Developers authenticating to push/pull code
• Developers authenticating to evaluate/approve merge requests
• External system API access
• Webhooks
• Additionally for Hosted Repositories/Workflows
• GitHub: Apps and Actions
• GitLab: Integrations
• Both: External collaborators

18. Open Source Component Management

19. Open Source Component Management
• Are you:
• Directly pulling from Central repositories
• Proxying requests through a Remote repository
• Ability to enforce policies
• Known security vulnerabilities in open source components
• License restrictions on open source components

20. Open Source Backdoor Concerns
• PHP example
• https://arstechnica.com/gadgets/2021/03/hackers-backdoor-php-source-code-after-breaching-internal-git-server/
• Linux kernel example
• https://www.techrepublic.com/article/linux-kernel-security-uproar-what-some-people-missed/
• Not realistic to detect
• Maintainers don't even have the bandwidth
• Have to be able to respond
• Pulling from Remote vs directly from Central
• Policies on acceptable (and unacceptable) versions

21. Build Management

22. Build Management
• Combining source code and open source components to create a new
build
• How often run:
• Every merge request
• Accepted merge requests
• Specific branches/tags
• Maven plugins – yet another source of code you can't trust that runs
in your environment!

23. Automated Testing

24. Automated Testing
• Most common types of testing:
• Unit testing
• Acceptance testing
• Other common analysis:
• Code quality ("smells")
• Code metrics (complexity, etc)
• Code coverage

25. Automated Testing: Security Testing
• SAST
• What is being analyzed: source or binary
• For a given language: is "binary" even a thing?
• Where is the analysis being performed: on-network, in the cloud
• Where are the results being stored: on-network, in the cloud
• DAST
• Where is the server being tested: on-network, in the cloud
• Where is the test traffic being generated: on-network, in the cloud, proxied
onto-network
• Where are the results being storoed: on-network, in the cloud

26. Automated Testing: Security Testing (cont'd)
• IAST
• Where is the server being tested: on-network, in the cloud
• Where is the test traffic being generated: on-network, in the cloud,
proxied onto-network
• Where are the results being stored: on-network, in the cloud
• SCA
• Where is the analysis being performed: on-network, in the cloud
• Where are the results being stored: on-network, in the cloud

27. Security Testing and Backdoors
• Best backdoor: Normal-looking security vulnerability
• "Oops, sorry I introduced that [SQL injection|missing auth check|etc]
• Other types of backdoor detection
• Static analysis – see the theoretical behavior of the application
• Look for suspicious behavior patterns
• References:
• https://www.acsac.org/2007/workshop/Wysopal.pdf
• https://www.veracode.com/sites/default/files/Resources/Whitepapers/static-detection-of-
backdoors-1.0.pdf
• https://owasp.org/www-pdf-archive/Protecting_the_Enterprise_-_Software_Backdoors.pdf
• https://owasp.org/www-pdf-archive/Protecting_Your_Applications_From_Backdoors.pdf

28. Software Packaging and Distribution

29. Software Packaging
• Monolithic applications vs. Microservice applications
• Application binary vs. Application binaries combined with containers
• Source container management is very similar to open source
component management
• Code signing is important
• But signed malicious/vulnerable code is still malicious/vulnerable code

30. Software Distribution
• Now the software should be ready too distribute to customers
• Internal
• External
• How is the software distributed?
• Binary
• Binary + Container
• Binaries + Containers + orchestration
• How are builds verified?
• Checksums

31. Two Exercises
• Passive: Grab netflow data from your build environment and see
what it talks to
• Active: Follow code change from checkout to release
• Spoiler alert: the active exercise where I engaged the development
team was more productive

32. Reviewing Netflow Data
• Pull netflow logs from network where development infrastructure is hosted
• Look at what the build infrastructure talks to
• Code repositories
• Open source component repositories
• Testing services
• Other services <- Area of concern
• Container repositories
• In practice
• Identified some interesting communications
• "Expensive" to run down results
• "High" false positive rate

33. Netflow Data Review
[Run through cleaned up data from build infrastructure netflow review]

34. Follow a Code Change
• Change code
• Check out code (and make changes)
• Create merge request
• Review merge request
• Run build
• Pull code
• Pull open source components
• Automated testing
• Unit tests
• Acceptance tests
• Security tests
• Other tests
• Accept build
• Push to distribution
• Distribute software
Questions to ask at each stage
• How does this user/system authenticate to
the next step
• Where is the server/service that performs this
step
• Where possible: show, don't tell
• Look at the UI developers look at
• Ask questions about where data comes from

35. GitLab Walkthrough

36. Jenkins Walkthrough

37. GitHub Walkthrough

38. Parallel Build Pipelines
• Have two or more separate, parallel build pipelines
• Compare artifacts at the end
• Goal is to force attackers to compromise multiple pathways
• Best way to get budget for this is to have a public breach of epic
proportions and suggest that your successor try it out
• How do you compare the outputs from multiple toolchains?
• Compare SBOM
• Asset enumeration
• Reverse engineering
• Live/behaviral testing

39. Using the Threat Model
• Identify gaps in your own pipeline security
• What can you shore up?
• What do you have to live with?
• How do you respond to incidents?
• Have meaningful conversations with suppliers of your software
• Understand their practices
• Push for change – vote with your wallet
• Be better able to detect malicious behavior

40. Vendor Management

41. Questions