Today’s cutting edge companies have release cycles measured in days instead of months. This agility is enabled by the DevOps practice of continuous delivery, which automates building, testing, and deploying all code changes. This type of automation will help you catch bugs sooner and accelerate developer productivity. In this session we will share our AWS engineers embed security practices in DevOps, and discuss how you can use AWS services to securely enable DevOps agility in your organization.

1. © 2016 AWS and affiliates, all rights reserved
Introduction to DevSecOps on AWS

2. © 2016 AWS and affiliates, all rights reserved
What is DevOps?
Cultural
Philosophy
Practices Tools

3. © 2016 AWS and affiliates, all rights reserved
Competing Forces
Business
Development Operations
Build it faster Keep it stable
Security
Make it
secure

4. © 2016 AWS and affiliates, all rights reserved
Who is DevSecOps
DevSecOps is
• Team/Community effort, not a person
• Automated and autonomous security
• Security at scale
DevSecOps role
• Not there to audit code
• Implement the control segments to validate and audit code and artifacts as part of
the CI/CD process
Security
OperationsDevelopment

5. © 2016 AWS and affiliates, all rights reserved
Terminology
DevSecOps / Security Automation / Security at Scale
Make up your mind…

6. © 2016 AWS and affiliates, all rights reserved
Why security automation
Reduce risk of human error
- Automation is effective
- Automation is reliable
- Automation is scalable

7. © 2016 AWS and affiliates, all rights reserved
Why security automation
Reduce risk of human error
- Automation is effective
- Automation is reliable
- Automation is scalable
Don’t worry…we still need humans

8. © 2016 AWS and affiliates, all rights reserved
Why security automation
High pace of innovation is great

9. © 2016 AWS and affiliates, all rights reserved
Why security automation
We also want high pace of:
• Detection
• Alerting
• Remediation
• Countermeasures
• Forensics

10. © 2016 AWS and affiliates, all rights reserved
What is DevSecOps
Three flavors
- Security of the CI/CD Pipeline
- Automated IAM roles, Jenkins server hardening, etc.
- Security in the CI/CD Pipeline
- Automated security tests, code analysis, etc.
- Security Automation
- Automated Incident Response Remediation, forensics, etc.

11. © 2016 AWS and affiliates, all rights reserved
Security in/of the CI/CD Pipeline

12. © 2016 AWS and affiliates, all rights reserved
What is DevSecOps
DevOps = Efficiencies that speed up this lifecycle
DevSecOps = Validate building blocks without slowing lifecycle
developers customers
releasetestbuild
plan monitor
delivery pipeline
feedback loop
Software development lifecycle
Security

13. © 2016 AWS and affiliates, all rights reserved
CI/CD for DevOps
Version
Control
CI Server
Package
Builder
Deploy
Server
Commit to
Git/masterDev
Get /
Pull
Code
AMIs
Send Build Report to Dev
Stop everything if build failed
Distributed Builds
Run Tests in parallel
Staging Env
Test Env
Code
Config
Tests
Prod Env
Push
Config
Install
Create
Repo
CloudFormation
Templates for Environment
Generate

14. © 2016 AWS and affiliates, all rights reserved
Version
Control
CI Server
Package
Builder
Promote
Process
Validate
Git-SecretsDev
Get /
Pull
Code
AMIs
Log for audit
Staging Env
Test Env
Code
Config
Tests
Prod Env
Audit/Validate
Config
Checksum
Continuous
Scan
CI/CD for DevSecOps
Send Build Report to Security
Stop everything if audit/validation failed
CloudFormation
Templates for Environment

15. © 2016 AWS and affiliates, all rights reserved
What Does DevSecOps CI/CD Give Us?
• Confidence that our code is validated against corporate security
policies.
• Avoid infrastructure/application failure in a later deployment due to
different security configuration
• Match DevOps pace of innovation
• Audit and alert
• Security at scale!

16. © 2016 AWS and affiliates, all rights reserved
Security Automation

17. © 2016 AWS and affiliates, all rights reserved
AWS CloudFormation primer
Infrastructure is code

18. © 2016 AWS and affiliates, all rights reserved
AWS CloudFormation Primer
Allows you to define a “template”
• Composed of different “resources”
• Provision that template into repeatable, live, “stacks”.
CloudFormation (CFn) provides a single service interface
• Let CFn perform state changes and govern who calls CFn
Treat as Code
• Check in your templates
CFn templates can hook into external configuration management frameworks
• Jenkins/Chef/Puppet/etc.

19. © 2016 AWS and affiliates, all rights reserved
AWS CloudFormation Stacks
JSON/YAML
Template
Stack Stack Stack
Dev
Test
Staging
Prod
Demos
Regions

20. © 2016 AWS and affiliates, all rights reserved
Split Ownership Configurations
Who knows your solution best?
• Dev, Infra, Sec…?
• Delegate ownership
• Infra – VPC design, IGW Deployment, Subnets, etc
• DevOps – EC2, Elastic BeanStalk, RDS, DynamoDB, etc
• OS Patching, Security Agent Deployments, IAM Roles, etc
Use Yaml and split file into chunks or functions
• Separate file sources with access control – Use IAM/VPC-E/etc.
• Push files -> Validate -> Merge files -> Validate -> Deploy -> Validate
AWS CodePipeline or Jenkins for deployment
• Promotion flows
• Move from manual to Automation based on validation quality
• Excellent for merging jobs of split configurations

21. © 2016 AWS and affiliates, all rights reserved
Merging
From single file or multiple files
• Maintain access control using policies
• Use different source stores if needed
Based on function/state
Reusable patterns
Maintain order, especially of validation
• Security validation last to execute
• Security should always win

22. © 2016 AWS and affiliates, all rights reserved
Validation
Keep track of what section
you are validating
• Stage vs Prod
• Merged vs separated
Validate often and log/alert
• Validate part and end result
• Run-time validation
Tools
• AWS CodeCommit
• AWS CodeBuild
• AWS Lambda
• Config / Config Rules
• CloudWatch Logs / CloudWatch Events
• Etc.

23. © 2016 AWS and affiliates, all rights reserved
Where else can this be applied?
CloudFormation
Template
Task Definition Application
Specification File
(AppSpec file)
…and more.
AWS CloudFormation AWS CodeDeployAmazon EC2 Container Service

24. © 2016 AWS and affiliates, all rights reserved
AWS Tooling
Execution
• Lambda
Tracking
• AWS Config Rules
• Amazon CloudWatch Events
• AWS CloudTrail
• AWS Inspector
Track/Log
• Amazon CloudWatch Logs
• Amazon DynamoDB
Alert
• SNS
Third party Open Source

25. © 2016 AWS and affiliates, all rights reserved
Other resources / Open Source
• Some of the projects out there:
– ThreatResponse.cloud https://threatresponse.cloud
– Cloud Custodian https://github.com/capitalone/cloud-custodian
– Security Monkey https://github.com/Netflix/security_monkey
– FIDO https://github.com/Netflix/Fido
• And many more…

26. © 2016 AWS and affiliates, all rights reserved
Automatic Incident Response
Remediation

27. © 2016 AWS and affiliates, all rights reserved
Creating a blueprint
Continuous /
Event based
Config Rules
CloudWatch
Events
Is it region
specific
Will action risk
breaking
something
Yes: Call human
No: Lambda
Will enable add
cost
Yes: Based on
possible cost
limit call
human
No/Minor: Set
rules
Is there a source
of truth
Config Rules:
Check previous
•Caution on
multiple events
CWE: Check
tag/DDB
•Have default
value
Action
Revert change
based on above
Forensic
Is it human (or
unknown
source) or
machine
(CI/CD)
CI/CD: Create
ticket (Jira etc)
Human: Should
we
countermeasur
e/prevent?
Are they using
MFA
•No: Add MFA
(external Lambda)
First occurrence
(check DDB)
•Yes: Disable
account/Keys
Alert
High:
SMS/Page
Low:
Email/tracking
system
Logging
Is it sensitive
Yes: Encrypt
(KMS)
No: Cleartext
Always: Access
control

28. © 2016 AWS and affiliates, all rights reserved
The anatomy of security automation
Mode
Section Actions
Initiate
React Config Rules / CloudWatch Events / Log Parsing
Trigger Lambda
Learn Lambda / CloudWatch Logs
Execution
Priority Action Restart service, delete user, etc.
Forensics Discover: Who/where/when, allowed to execute?
Countermeasure Disable access keys, isolate instance, etc.
Alert Text/Page, email, ticket system
Logging Database, ticket system, encrypt data?

29. © 2016 AWS and affiliates, all rights reserved
How do I know what happened - Config

30. © 2016 AWS and affiliates, all rights reserved
The key to Custom Rules
response = client.put_evaluations(
Evaluations=[
{
'ComplianceResourceType': 'string',
'ComplianceResourceId': 'string',
'ComplianceType': 'COMPLIANT'|'NON_COMPLIANT'|'NOT_APPLICABLE'|'INSUFFICIENT_DATA',
'Annotation': 'string',
'OrderingTimestamp': datetime(2015, 1, 1) },
],
ResultToken='string’
)

31. © 2016 AWS and affiliates, all rights reserved
How do I know what happened – CloudWatch Events
{
”account”: “111111111111”,
”region”: “us-east-1”,
”detail”: {
”eventVersion”: “1.02”,
”eventID”: “c78ce8de-46ee-4fea-bcf4-0e889d419f2f”,
”eventTime”: “2016-01-18T03:32:18Z”,
”requestParameters”: {
”userName”: “trigger”
},
”eventType”: “AwsApiCall”,
”responseElements”: {
”user”: {
”userName”: “trigger”,
”path”: “/”,
”createDate”: “Jan 18, 2016 3:32:18 AM”,
”userId”: “AIDAIKL7LKTAUFPNJQ3LY”,
”arn”: “arn:aws:iam::111111111111:user/trigger”
}
},
”awsRegion”: “us-east-1”,
”eventName”: “CreateUser”,
”userIdentity”: {
”userName”: “IAM-API-RW”,
”principalId”: “AIDAI5SJPHVGH1WK7HTQS”,
”accessKeyId”: “AKIAIGYEYSX4EVED52YA”,
”type”: “IAMUser”,
”arn”: “arn:aws:iam::111111111111:user/IAM-API-RW”,
”accountId”: “111111111111”
},
”eventSource”: “iam.amazonaws.com”,
”requestID”: “13bb2739-bd94-11e5-9abd-af4e7ff9090f”,
”userAgent”: “aws-cli/1.9.20 Python/2.7.10 Darwin/15.2.0
botocore/1.3.20”,
”sourceIPAddress”: “111.112.113.114”
},
”detail-type”: “AWS API Call via CloudTrail”,
”source”: “aws.iam”,
”version”: “0”,
”time”: “2016-01-18T03:32:18Z”,
”id”: “d818DD19-7b16-4e1d-a491-794a26b51657”,

32. © 2016 AWS and affiliates, all rights reserved
Different sources have different event
”eventName”: “CreateUser”,
”userIdentity”: {
”userName”: “IAM-API-RW”,
”principalId”: “AIDAI5RTPJGHE43K7GEQS”,
”accessKeyId”: “AKIADSJGHSXRKVDM52DA”,
”type”: “IAMUser”,
”arn”: “arn:aws:iam::111111111111:user
”accountId”: “111111111111”
”eventName”: “CreateUser”,
"userIdentity": {
"principalId": "AROGKTYFTCBFKTESCEVK:henrikj",
"accessKeyId": ”GFSHKUOLZG53JE5DHKRC",
"sessionContext": {
"sessionIssuer": {
"userName": ”AssumeAdministrator",
"type": "Role",
"arn": "arn:aws:iam::111111111111:role/Administrator",
"principalId": "AROSKTRDFTXBUFLSKCEVK",
"accountId": " 111111111111 "
},
"attributes": {
"creationDate": "2016-01-18T16:50:04Z",
"mfaAuthenticated": "false"
}
},
"type": "AssumedRole",
"arn": "arn:aws:sts::111111111111:assumed-
role/Administrator/henrikj",
"accountId": "111111111111"

33. © 2016 AWS and affiliates, all rights reserved
How can I get the different events?
import json
def lambda_handler(event, context):
eventdump = json.dumps(event, indent=2)
print("Received event: " + json.dumps(event, indent=2))
return eventdump

34. © 2016 AWS and affiliates, all rights reserved
Risks
• You can now automatically mess up your approved changes
• No proper alerting and follow-up on automatic events
• Over/under complicated scripts
• No info on desired state
• Race the hacker…automation wars!

35. © 2016 AWS and affiliates, all rights reserved
Best practices
Implement “Compliance Status” for easy overview
• Use pre defined checks
• Create extended custom checks
• Fix the issue while checking
Evaluate/remediate changes/events in your account
• Doesn’t replace log analysis (Machine Learning FTW)
• Protect against changes made by (un)authorized accounts
• Automatic remediation for critical events
• Do forensic on the fly
Always Log and Alert!