Manage Infrastructure Securely at Scale and Eliminate Operational Risks - DEV335 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
M a n a g e I n f r a s t r u c t u r e S e c u r e l y a t S c a l e a n d E l i m i n a t e
O p e r a t i o n a l R i s k s
A n a n t h V a i d y a n a t h a n – S e n i o r P r o d u c t M a n a g e r , A W S
D a n i e l K o o – S e n i o r D i r e c t o r , F I N R A
D E V 3 3 5
N o v e m b e r 3 0 , 2 0 1 7

What to Expect in this Session
• Introduction to AWS Systems Manager
• Solving key Enterprise IT problems
• Learn how FINRA uses Systems Manager
• Demo!
• Best Practices for using Systems Manager
• Q&A

Key Customer Problems—How do I…?
• Make configuration changes and adhere to security lockdown policies
• Get a holistic view of applications and OS settings with change
tracking
• Automate patching of Windows and Linux instances on AWS and on-
premises with a single cost-effective tool
• Store and retrieve secrets in my workflows in an automated and
secure manner

AWS Systems Manager
Group
• Create groups to reflect an application stack or
an environment
Visualize
• Centralize operational data from AWS services
• View patch compliance, audit data and more
Take
Action
• Automate remedial actions on groups
• Operate safely across resources
Manage on AWS
or on-premises
Natively works
with other AWS
services
Cross-platform
Available at no additional charge!

Learn How to Manage and Operate
Your Environments at Scale

Enterprise IT Problems
• Safe and secure operations on instances
• Understand what’s installed on my instances
• Inefficient and complicated patching processes
• Secure access and management of all my secrets

Safe and Secure Operations with RBAC
• Configuration changes without SSH-access
or bastion hosts
• Make changes at scale across groups of
instances
• Automate RBAC to set who can perform
what actions on which set of instances
• Control blast radius using rate control for
safety at scale
• Audit what actions were made on which
instance
VPC2
Corp data
center
VPC1
Tags
CloudTrail
Auditing
IAM
Tags
Amazon
CloudWatch
Events
Ansible
Playbook
Run
Command

Run Command – SSH-less Operations
Sample Use Cases:
• Take VSS consistent snapshots
• Collect logs from terminating instances in an Auto Scaling Group
• System monitoring and health checks
• Install applications or make registry edits
aws ssm send-command
--document-name HealthCheck –-document version $DEFAULT
--targets “Key=tag:Env,Values=Dev,Prod” “Key=tag:Role;Values=WebFrontEnd”
--max-concurrency 10
--max-errors 20
--service-role-arn <my-service-role>
--notification-config NotificationArn=<SNS Topic
ARN>,NotificationEvents=“Success”,NotificationType=“Command”

Pre-announcing Instance Health Monitoring
• Cross-platform metrics and logs monitoring using Amazon
CloudWatch and Systems Manager
• Supports high-resolution metrics with aggregation
• Metrics such as CPU, Memory, Disk, Custom, and logs
• Simplified setup
• Open-sourced framework

Run Command - Secure Environment
{
"Effect": “Allow",
"Action": [ "ssm:SendCommand“ ],
"Resource":
"arn:aws:ec2:<region>:<account>:instance/*",
"Condition": {
"StringLike": {
"ssm:resourceTag/App":
[“WebServer"]
...
{
"Effect": "Allow",
"Action": [ "ssm:SendCommand" ],
"Resource":
"arn:aws:ssm:<region>::document/*“,
"Condition": {
"StringLike": {
"ssm:resourceTag/WebTeam":
[“OnlyConfigWebServer"]
...
Access controlled instances,
buttoned up environment!

SSM Document – Management as Code
• Automate repetitive IT tasks and operations through code
• Announcing YAML support for authoring configurations!
• Works cross-platform across Windows and Linux
• Centralized management through document sharing across accounts to
enforce best practices
• Run from external locations such as public or private GitHub repositories

SSM Document – Authoring
---
schemaVersion: "2.2"
description: "MyConfigurationAsCode"
mainSteps:
- action: "aws:runDocument"
name: "InstallApache"
inputs:
documentType: "SSMDocument"
documentPath: "arn:aws:ssm:us-west-1::document/InstallApache"
documentParameters: "{ }"
- action: "aws:downloadContent"
name: "AnsiblePlaybookfromGitHub"
inputs:
sourceType: "GitHub"
sourceInfo: "{"owner":"AnanthV", "repository":"MyPrivateRepo","path"
:"documents/nginx.yml", "tokenInfo":"{{ ssm-secure:MyAccessToken }}"}"
destinationPath: ""
- action: "aws:runShellScript"
name: "RunPlaybook"
inputs:
commands:
- "ansible-playbook -i "localhost," -c local nginx.yml"
Nested Document
Download Ansible Playbook from
my private GitHub repo
Run the Ansible Playbook

Holistic View of What’s on Your Instances
• Collect software inventory such as
applications, system properties or custom
attributes
• Create inventory data lake on Amazon S3
• Query inventory across accounts and
regions
• Gain insights by building analytics and
visualizations
• Track changes for auditing and compliance
Account 2
Corp data center
Amazon
Athena queriesAmazon S3
Data Lake
Account 1
BI Tools
Amazon
QuickSight
AWS Config
Inventory

Inventory—Collect & Query Software Information
• Announcing Inventory dashboard with
aggregations
• Pre-defined types such as Applications,
Network Information, Files, Windows
Servers/Roles
• Collect custom Inventory as per business
needs
• Historical record of Inventory changes using
AWS Config
• Use Resource Data Sync to collect cross-
account/region inventory to a single S3
bucket
--------- Custom Inventory -----------
aws ssm put-inventory --instance-id "ID" -
-items '[{"CaptureTime": "2016-08-
22T10:01:01Z", "TypeName":
"Custom:RackInfo",
"Content":[{"RackLocation": "Bay B/Row
C/Rack D/Shelf E"}], "SchemaVersion":
"1.0"}]‘

Recap—Solving Key Enterprise Problems
ü Safe and secure operations without SSH or bastion hosts
ü Run Command
ü SSM Documents
ü Holistic view of what is installed on your instances
ü Inventory
ü Resource Data Sync
• Automated patching of Windows and Linux instances with compliance
reporting
• Centralized store for secrets with easy access control and retrieval

OS Vulnerability Compliance & Prevention
• Rules: Automate the approval of patches
using patch baselines
• Apply Patches: Use Maintenance Windows
on Windows and Linux instances
• Report: Compliance reporting for audit and
remediation
• Remediate: Quickly apply patches to fix a
zero-day vulnerability such as Petya
Corp data center
Patch Group = DBServers
Patch Group = WebServers
Patch Group = SQLCluster
DB Server
Patch Baseline
Web Server Patch
Baseline
Patch Manager
Maintenance
Window
Compliance
Notifications!

Patch Manager and Compliance
Patch
Rules
1 instance
Critical non-
compliant
Missing
Patches with
Severity

Centralized Store for Secrets
• Centralized store for all secrets such as
passwords and API keys
• Organize parameters into hierarchies
• RBAC for parameters or hierarchies at any
API level
• Retrieve and use across AWS services. Only
need access to AWS CLI or APIs!
• Track who used a parameter and when to
maintain an audit trail
Dev Test Prod
App App App
/test/app/db_password /prod/app/db_password
email notification
Rotate password
event
(event-based)

Demo

Learn How FINRA Uses Systems Manager
D a n i e l K o o – S e n i o r D i r e c t o r , F I N R A

About FINRA
Investor
protection
Market
integrity

FINRA: Volume
brokers
12
firms
3,800 634,000
markets/
exchanges
37 billion
events on average
each day

FINRA: Big Data
of storageevents per day
20+pb 100sof nodes and
edges
of complex
queries
trillion75 billion
Up to
5000+
running instances
150+
applications

Problems
TransparencyGovernance
DevOps
Access Control Transient Platform
Compliance

Gatekeeper Application
• Homegrown web application
• Access management system
• Grant temp access to EC2 (Linux &
Windows)
• Self-serviced model with approval
process
Rundeck SSM Plugin
• Plugin for Rundeck open source tool
• Used to execute Ops jobs
• Runs across a fleet of instances
• Eliminates the need for SSH
Current Solutions

Gatekeeper
Gatekeeper
App
Users
Call SSM
on VPC
Store Request
Data
EC2
Search EC2 &
AWS API
SSM
AWS VPCs

{
"schemaVersion":"1.2",
"description":"Script for GateKeeper to create temp user.",
"parameters":{
… Parameter details here …
},
"runtimeConfig":{
"aws:runShellScript":{
"properties":[
{
"id":"0.aws:runShellScript",
"runCommand":[ "useradd -e `date -d '+2 days' '+%Y-%m-%d'` {{ userName }}",
"mkdir /home/{{ userName }}/.ssh",
"echo '{{ publicKey }}' >> /home/{{ userName }}/.ssh/authorized_keys",
"chown -R {{ userName }}:{{ userName }} /home/{{ userName }}",
"chmod -R go-rwx /home/{{ userName }}/.ssh",
"echo '{{ userName }} ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/{{ userName }}" ],
"workingDirectory":"/root",
"timeoutSeconds":"{{ executionTimeout }}"
}
]…
SSM Document—Create User

{
"schemaVersion":"1.2",
"description":"Script for GateKeeper to cleanup expired users.",
"parameters":{
… Parameter details here ...
},
"runtimeConfig":{
"aws:runShellScript":{
"properties":[
{
"id":"0.aws:runShellScript",
"runCommand":[ "cut -f1 -d':' /etc/passwd | grep {{ userName }} > /dev/null && (userdel -rf {{
userName }} ; echo 'user deleted' ) || echo 'no user to delete'",
"ls /etc/sudoers.d/ | grep {{ userName }} > /dev/null && (rm -f /etc/sudoers.d/{{
userName }} ; echo 'sudo file deleted' ) || echo 'no sudo file to delete'" ],
"workingDirectory":"/root",
"timeoutSeconds":"{{ executionTimeout }}"
}
]…
SSM Document—Remove User

Benefits
Before
• Share PEM files (less secure)
• Management nightmare (permanent
users on instances)
• Limited transparency
• Slow, error-prone
• High cost (people, manual, tools)
After
• Temporary PEM files (more secure)
• Simple user management (short lived
temp users on instances)
• Maximum transparency
• Fast, reliable, automated
• Low cost (self-serviced, SSM is free)

Rundeck SSM Plug-in
Amazon S3
Rundeck
SSM
Service
AWS VPC
AWS VPC
AWS VPC
EC2
EC2

How It Works
• Running command or executing script on the instance
• Rundeck makes API call to SSM
• SSM executes on target instance
• When SSM finishes, results are collected and sent back to Rundeck
• Executing script not staged on the instance
• Rundeck uploads script to S3
• Rundeck makes API call to SSM
• SSM executes on target instance to download script from S3 and run
• When SSM finishes, results are collected and sent back to Rundeck

Benefits
Before
• Sharing of SSH keys (less secure)
• Complex network setup (firewalls,
security groups)
• Difficult to scale
• Limited reliability
• Expensive
After
• No SSH, use IAM policies (more secure)
• Simplified network topology (instances
only need access to S3 and SSM)
• Highly scalable (multiple accounts,
regions)
• Fast and reliable
• Low cost

Upcoming
• Parameter Store
• Fetch secrets during deployment using AWS services (CodeBuild/Pipeline/…)
• Retrieve secrets at run time via API
• State Manager
• Windows Server configuration automation
• Collect compliance report on server configuration
• Auto-remediation of non-compliant servers
• Patch Manager / Automation
• Ad-hoc patching capability on running instances for critical vulnerabilities
• Collect compliance report on patches applied on instances

Demo
Request temporary access to EC2 via Gatekeeper
Approve access request
Send temporary access key
Login to EC2 using temporary key
Expire temporary access

Recap and Best Practices
• Systems Manager (SSM) is a platform to automate Enterprise IT operations
safely and securely
• SSM integrates with AWS services such as IAM, CloudTrail, CloudWatch
Events, Amazon Config to provide automation, visibility
• Available in all AWS regions including GovCloud, and accessible through
AWS PrivateLink
• SSM Agent is on by default on Windows Server and Amazon Linux AMIs
• Use SSM to update agent regularly to avail of new features
• SSM is SOC and HIPAA certified

Learn More About Systems Manager
• Learn more—https://aws.amazon.com/ec2/systems-manager/
• AWS Blog—https://aws.amazon.com/blogs/aws/category/amazon-ec2-
systems-manager/
• AWS Management Tools Blog—
https://aws.amazon.com/blogs/mt/category/management-
tools/amazon-ec2-systems-manager/
• DEV306—Embrace DevOps and learn how to Automate Operations
(12/1/2017 at 9:15 AM)
• Feedback—ec2-ssm-feedback@amazon.com

Thank you!

Manage Infrastructure Securely at Scale and Eliminate Operational Risks - DEV335 - re:Invent 2017

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Manage Infrastructure Securely at Scale and Eliminate Operational Risks - DEV335 - re:Invent 2017

Similar a Manage Infrastructure Securely at Scale and Eliminate Operational Risks - DEV335 - re:Invent 2017 (20)

Más de Amazon Web Services

Más de Amazon Web Services (20)

Manage Infrastructure Securely at Scale and Eliminate Operational Risks - DEV335 - re:Invent 2017