CI/CD pipelines are quickly becoming the path of least resistance for would-be attackers into sensitive internal systems, gaining access to critical data, with minimal effort.
In the InfoSec world when we talk about CI/CD security often times this focuses on specific aspects of securing your pipeline - scanning the code, protecting secrets, securely managing code deployments, or even authentication and authorization mechanisms, but we rarely talk about all of these together.
After years of being in the trenches and realizing that the attack surface is growing and the threat landscape becoming more and more complex, it has become increasingly apparent that security teams need to adapt and modify strategies to keep up with the new reality of CI/CD protection, without compromising developer velocity.
In this talk I would like to propose a new way of thinking about CI/CD security - that encompasses the three disciplines that comprise CI/CD security - security in the pipeline, of the pipeline, and around the pipeline. Partial coverage of any or all of these disciplines simply will not cut it with the continuously evolving risk landscape. Security engineers need to address each of these aspects in their entirety to provide the full scope of coverage that modern organizations need, and I will take a deep dive on the challenges each introduce, and the approaches and techniques for mitigating them based on adversarial sec research.
5. How well is security adapting to
these changes?
6. cidersecurity.io
The Engineering Ecosystem
Repo CI Pipeline CD Pipeline
Artifact
Language
SCM CI CD
Artifact
Repository
Container
Registry
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
7. cidersecurity.io
The Challenge
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
Repo
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CI Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
CD Pipeline
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Artifact
Language
Language
Language
Language
Language
Language
Language
Language
Language
Language
Language
Language
SCM CI CD
Artifact
Repository
Container
Registry
SCM CI CD
Artifact
Repository
Container
Registry
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator Collaborator
Collaborator
Collaborator
Collaborator
Repo
Repo
Artifact
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
Collaborator
8. cidersecurity.io
Github Jenkins Artifactory
ECR
EKS
The complexity
User 1 User 3
User 2 User 4 App 1 App 3
App 2 App 4
Devops Repo
Service 1 Repo
Service 2 Repo
Terraform
Pulumi
Jenkinsfile
Python
JavaScript
Jenkinsfile
Ruby
JavaScript
Jenkinsfile
Artifact 1
Artifact 2
Artifact 3
Artifact 4
Pipeline 3 (CD)
Deploy{...}
Artifactory_Read_Key
AWS_Access_Key_1
AWS_Access_Key_2
Pipeline 1 (CI)
Build {...}
Test {...}
Pipeline 2 (CI + CD)
Build {...}
Test {...}
Deploy{...}
Artifactory_write_key
R
RW
Pod 1 Pod 2
Container 2
Container 1
10. cidersecurity.io
Engineering environments have become
the new attacker’s turf
Today’s attack surface
A single insecure step in the CI, or
insecure package import - can lead to
devastating results
Engineers are also looking for ways to
bridge the gap
14. cidersecurity.io
Github
Devops Repo
Service 1 Repo
Service 2 Repo
Gitlab
Devops Repo
Service 3 Repo
Service 4 Repo
Terraform
Pulumi
Jenkinsfile
Python
JavaScript
Jenkinsfile
Ruby
JavaScript
Jenkinsfile
Ansible
Chef
Python
Java
Go
JavaScript
Jenkinsfile
SIP - Security In the Pipeline
15. cidersecurity.io
Github
Devops Repo
Service 1 Repo
Service 2 Repo
Gitlab
Devops Repo
Service 3 Repo
Service 4 Repo
Terraform
Pulumi
Jenkinsfile
Python
JavaScript
Jenkinsfile
Ruby
JavaScript
Jenkinsfile
Ansible
Chef
Python
Java
Go
JavaScript
Jenkinsfile
SIP - Security In the Pipeline
16. cidersecurity.io
Scanner Issue Description Severity Repo Location
Checkov Bad stuff Extremely Bad Repo 1 Line 1
GoSec Bad stuff Horrible Repo 2 Line 2
Bandit Bad stuff Very severe Repo 1 Line 4
Brakeman Bad stuff Not good Repo 3 Line 5
Checkov Bad stuff Fix now Repo 4 Line 2
PMD Bad stuff Fix fast Repo 1 Line 3
Nodejsscan Bad stuff So so Repo 2 Line 7
Nodejsscan Bad stuff doing ok Repo 3 Line 18
17. cidersecurity.io
SIP/SOP/SAP
Comprehensive Technical DNA of your environment -
from Code to Deployment
SIP - Security In
the Pipeline
Addresses the risk
of code with
security flaws
flowing through the
pipeline
SOP - Security Of
the Pipeline
Addresses the risk
of the systems in
the pipeline being
compromised
19. cidersecurity.io
SIP/SOP/SAP
Comprehensive Technical DNA of your environment -
from Code to Deployment
SIP - Security In
the Pipeline
Addresses the risk
of code with
security flaws
flowing through the
pipeline
SOP - Security Of
the Pipeline
Addresses the risk
of the systems in
the pipeline being
compromised
SAP - Security
Around the Pipeline:
Addresses the risk
of the pipeline being
bypassed
21. cidersecurity.io
SIP/SOP/SAP
Comprehensive Technical DNA of your environment -
from Code to Deployment
SIP - Security In
the Pipeline
Addresses the risk
of code with
security flaws
flowing through the
pipeline
SOP - Security Of
the Pipeline
Addresses the risk
of the systems in
the pipeline being
compromised
SAP - Security
Around the Pipeline:
Addresses the risk
of the pipeline being
bypassed
22. cidersecurity.io November 2021
Takeaway #1 - for
defenders
●Appsec has extended far
beyond the scope of code
scanning.
●To address today’s
challenges, we need to be
thinking about SIP, SOP
and SAP
Good morning everyone,
this is the 3 disciplines of CI/CD security
Great honor and privileged to be speaking here
A short intro
I’m the Co-Founder and CTO of Cider
Many people ask us why ‘what kind of a name is Cider’
Releases are conducted on a daily or hourly basis
The stack is comprised of more technologies
It takes a much shorter time to adopt a new technology or framework
Not only applications are codified, infra is
And of course a lot less manual process, a lot more automation and continuous integration continuous delivery
Security is struggling to keep up
Especially relevant now that security is no longer a blocker
when we look at the The building blocks that comprise the ecosystem
Different systems
moving around different types of objects and artifacts
all the way from the engineers endpoint to production
a fusion of human collaborators and services and applications accessing the systems
and a lot of 3rd parties and access tokens and keys spread out through the environment
And the challenge for us as defenders,
Is that reality doesn’t really like in the slide we saw earlier
it looks a little more like this
Even in a small startup, definitely a big organization
Each one of these building blocks being potentially connected to one or more of the others
And the complexity of coping with the challange,
stems from how deeply familiar we need to become with the inner working of the environment
in order to understand where the risks are and what security measures are required ((click))
what repos
what languages
how do CI pipelines connect to repos, with what permissions
what secrets are stored in CI and what is their scope
how do CI and CD pipelines take code, package it,
upload to artifcat repositories, which are then bundled in cotaniers
and ultimately deployed to prod
and which humans and which applications have access to that ecosystem
unless we know all of this, it is pretty hard to understand what security risks exist in our ecosystem
Coping with the challange of what is going on in this fast paced and dynamic ecosystem
Very easy to get lost
Have partial visibility and understanding of what’s going
Don’t really know who to refer to if we have specific questions
In parallel what’s evident is that engineering have become a primary area of focus for attackers
some examples of that, which I’m sure many of you are familiar with - just from the past year -
Solarwinds - which had their built/ci system compromised, ending with malware being shipped to 18000 orgs
the codecov hack - where orgs using codecov as part of their ci had their environment variables
php - that had their git infrastructure compromised and served a PHP version with a backdoor
dependency confusion - where apple, msft and dozens of other giants were at the risk of having their CI compromised by managing dependencies in an insecure manner
and the recent COA, RC, UA PARSES NPM packages - with millions of weekly downloads, were compromised and infected with malware
So in this reality - it’s not just about security bridging the gap towards engineering
We at Cider have defined 3 disciplines which, together, help organizations
address the challenges and complexities we described earlier and build strong CI/CD security progrmas
they are called SIP, SOP, SAP
Security in the pipeline
Security of the pipeline
Security around the pipeline
In the next slides we’ll review each one and understand what they are
We have to keep in mind that , as we discussed, building strong CI/CD security programs requires us to begin a very intimate level of familiarity with the ecosystem,
the technologies and the interconnectivity between the different systems.
Having that “technical DNA” is basically our base layer on top of which we build our CI/CD security program
let’s start with SIP
Security in the pipeline is about implementing the effective measure to detect security flaws in our code
CI Pipeline
CI Pipeline
CI Pipeline
Security of the pipeline is about understanding that hackers are targeting our SCM, our CI, the rest of the systems down the pipeline, and we need to make sure they are secure enough to prevent those attacks
The best way to understand SOP is to look at it from the attacker’s perspective,
and that means looking at it from the crown jewels
SOP is the equivalent of the solutions in blue, for the vector of abusing software delivery systems and processes for getting to production
SAP - addresses the concern of our pipeline being bypassed
So when we look at our ecosystem...
If we think about it,
it’s not enough to be perfect in SIP and in SOP, if someone can connect directly to k8s and deploy malware to production. Or connect directly to AWS and modify a lambda function in a manner that isn’t consistent with what’s stored in our SCM.
(click)
So SAP is about taking the measure to be able to answer 2 main questions
SAP - addresses the concern of our pipeline being bypassed
Daniel and I are coming from many years in the trenches and we felt the pain in our day-to-day work. Even companies like AppsFlyer that have strong security teams and strong understanding of the need for security as part of engineering (it was a competitive advantage for us) can’t handle the complexity and the security teams are struggling with working with the engineering teams. I always felt that I have so many blind spots in the engineering even after I found something I realized that there are many other issues that I’m not aware.
We need to change the way we interact with engineering teams. We need to do it better and faster.
This is why we established Cider.