Large AWS environments have assets distributed across many accounts and regions. Ideally, asset inventory should be timely and provide an audit trail to document who made the changes and when. This is required for security teams to quickly react to insecure configurations and for DevOps tooling to manage infrastructure effectively. The traditional means of obtaining the timely and current state of AWS assets is to very frequently poll over the entire infrastructure, often tens of times per minute. This becomes increasingly difficult as AWS infrastructures grow in complexity. Additionally, polling for infrastructure changes provides no auditability context. In this session, learn how to inventory, track, and respond to AWS asset changes with seconds at scale.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inventory, Track, & Respond to
AWS Asset Changes within
Seconds at Scale
Mike Grima
Senior Cloud Security Engineer
S E C 3 9 1

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
What are we trying to solve?
Inventory Collection Approaches
Security Monkey
Tooling Challenges
Deep-dive into Historical
Desired AWS Infrastructure State

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowing is half the
battle
Infrastructure visibility is
extremely important
Need answers to the following
questions:
Which Amazon Web Services (AWS)
assets do I have deployed?
Where are these assets deployed?
Are they configured securely?
Did the configuration change?
Who created/updated/deleted a given
asset, and when?

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Follows Visibility
DFIR capabilities
Detection of compromise
Organizational policy violations
Identify publicly accessible resources
Detection of insecure-by-default configurations
Determination of resource ownership

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inventory collection requirements
Must be timely and accurate
Full-view of the environment
Attribution (i.e. AWS CloudTrail context)
Who/What/Where/When/How?
Change History

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to collect inventory?
Polling
Periodically “asking” AWS for the current state of the infrastructure
Events
AWS tells you when the infrastructure changes

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Polling pros vs. cons
Pros
Complete inventory
On-demand
Cons
Slow – bigger environments take longer to inventory
High Latency -Must be aggressive to reduce
Risk of rate limiting
No CloudTrail Context – mapping resources to CloudTrail entries is non-trivial

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event-driven pros vs. cons
Pros
Fast – react to changes when they happen (order of seconds)
Timely state of infrastructure
Enables fast response
*Can reduce rate-limiting
Cons
Complex – many moving parts required
Lossiness of events – a polling component is still required!

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Polling for security @ Netflix - Security Monkey
Originally open sourced in 2014 by Patrick Kelley
https://github.com/Netflix/security_monkey
Watchers describe infrastructure
Auditors check for insecure configuration
Reporters provide notifications (emails, etc.)

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some challenges
Netflix infrastructure in 2014 was very different than today
Then: Few AWS accounts, Roles, Security Groups, etc.
Now: Many accounts (100+), 1000+ Roles, Security Groups, etc.
Security Monkey ran on one instance for API, UI, and Watcher logic
Earlier this year it stopped working. Hard.
Very stale data – WEEKS old
Major refactor for 1.0 release in January 2018 (now over 40+ instances)

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Monkey – Refactoring

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some limitations
Slow at our scale
30+ min to poll for Security Groups, IAM Roles, etc. in just ONE account
Rate limiting
No CloudTrail context

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scalability challenges
Security Monkey is a great tool, but our infrastructure outgrew it.
Polling is not effective at large scale.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Back in time to 2016
Events are the way to go
Project “Banana Peel” – Security Monkey as an AWS Lambda
Stripped down version of Security Monkey
Made use of Amazon CloudWatch Events
Provided CloudTrail context and completed response in < 30 seconds!

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Banana Peel’s limitations
CloudWatch Event Buses didn’t exist yet
Complex orchestration for event centralization (back then)
Limit risks
Cumbersome
Data was “locked” into Security Monkey / closed-loop

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Historical
Serverless and Event-Driven
Tracks changes to AWS resources within seconds of a
change
Maintains the CloudTrail context of changes
Downstream consumers can subscribe to material
changes
https://github.com/Netflix-Skunkworks/historical

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Historical
🚧 👷 Hard-Hat required 👷 🚧
Under VERY active development
Historical inventories and monitors the environment
for changes
Flexible
NOT a closed-loop system

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Historical
Currently monitoring
Amazon Simple Storage Service (Amazon S3)
Amazon Elastic Compute Cloud (Amazon EC2) Security Groups
IAM is in active development and coming soon!
Roles
Groups
Users
Managed Policies
Server Certificates
Basic Terraform template for installation

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
General Concepts
CloudWatch Event tells us a resource changed
We describe that resource that changed
We check if the resource’s state is actually different from what we
previously knew about it
Noise reduction for tools that periodically overwrite configurations
If there was a change, we record the details, and provide downstream
consumers with the newest state of the resource

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
General Concepts
We immediately process CloudWatch Event notifications
We poll periodically to capture state in case an event is dropped
Event lossiness is very low
Every few hours – low priority
In region deployments for faster processing of events
Per-resource type stacks
Amazon S3, Amazon EC2 Security Groups, IAM, etc

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We ♥ Amazon SQS and Lambda
We make use of Amazon SQS to invoke Lambda functions as much as
possible.
It’s awesome:
Concurrency
Auto-scales
Retries without blocking
Dead-letter queue ability
Message delays

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Components – CloudWatch Event Buses

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Components – IAM Roles

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Poller
Periodically invoked every few hours
“Poller Tasker” schedules a Lambda
function (the ”Poller”) to list all assets in
an account and region

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Collector – Describes a resource
Poller Collector
Tasked by the Poller
Event Collector
Tasked by CloudWatch
Events
Resource config is
saved to the ”Current”
table
Cache of all existing items
for a given resource type

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Differ – Checks for actual changes
Changes to the Current table pass changes to the Differ
Proxied over via Amazon DynamoDB Streams to Amazon SQS for Differ invocation
Differ compares the new config with the previously known config stored
in the “Durable” table
Changes result in new change record
Non-changes are ignored

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Notification to downstream consumers
The “Durable” table’s
DynamoDB Stream invokes a
Proxy Lambda that pushes to
Amazon Simple Notification
Service (Amazon SNS)
The Proxy serializes the item
along with the CloudTrail
context in an easy-to-
consume JSON

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current Internal usage and stats
Source of truth for enabling Amazon S3 Server Access Logs for ALL S3
buckets at Netflix!
We have A LOT of S3 Buckets!
StreamAlert integration for Security Group alerting
Alert on open IP ingress rules, ANYWHERE in our environment!
Approx. 1 Minute from time of event to alert completion (vs. 15 minutes the old way)!

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Goals for Historical
Add more AWS resource types
Enhancements to
Make it easier to add new resource types
Increase speed!
Improve documentation
UI and API

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Monkey’s Future
A great tool that took us very far
Our infrastructure outgrew it
Bigger scale
Event driven (FAST!)
Better context
Reduce rate-limiting
Better downstream consumption
Support minor bug fixes
Community contributions always welcome

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Netflix’s Future AWS Infrastructure
Fast-response and auto-correcting
Discover bad signals; fix automatically
Prevent and undo-large scale automation failures

46. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mike Grima
LinkedIn & GitHub: mikegrima
mgrima@netflix.com

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.