YouTube recording: http://youtu.be/DWMfXH3OfoE
Getting started with Amazon Web Services (AWS) is fast and simple. These slides from our Best Practices webinar outline best practice guidance from many customers and the Amazon Web Services team, helping you gain advantage as your implement your projects in AWS. It also covers how you can ensure your applications are simple to manage, resilient and cost effective and how to set up accounts and use consolidated billing.
Handwritten Text Recognition for manuscripts and early printed texts
Journey through the Cloud - Best Practices Getting Started in the AWS Cloud
1. Best practices for getting
started with AWS
Ryan Shuttleworth – Technical Evangelist
@ryanAWS
2. Journey through the cloud
Common use cases & stepping stones into the AWS cloud
Learning from customer journeys
Best practices to bootstrap your projects
3. Best practices
Simple things to plan for when starting with AWS
Some technical and human considerations
Helping you put your best foot forward from the off
6. Choose use case that suits you
Low hanging fruit can be easiest way to ‘cut teeth’
7. Choose use case that suits you
Dev & Test
Spin environments up and down
on demand
Decouple development and test
environments from operations
constraints
Explore elasticity in a sandboxed
environment
Low hanging fruit can be easiest way to ‘cut teeth’
8. Choose use case that suits you
Dev & Test Backup & DR
Spin environments up and down Take part of your data or
on demand business applications step- by-
step into non-production DR use
Decouple development and test
environments from operations Understand cloud dynamics and
constraints test during controlled failovers
Explore elasticity in a sandboxed
environment
Low hanging fruit can be easiest way to ‘cut teeth’
9. Choose use case that suits you
Dev & Test Backup & DR Greenfield Project
Spin environments up and down Take part of your data or Embody best practice of cloud
on demand business applications step- by- computing in unconstrained
step into non-production DR use greenfield projects
Decouple development and test
environments from operations Understand cloud dynamics and Self contained web projects,
constraints test during controlled failovers document archiving etc
Explore elasticity in a sandboxed
environment
Low hanging fruit can be easiest way to ‘cut teeth’
10. Choose use case that suits you
Dev & Test Backup & DR Greenfield Project Pain point
Spin environments up and down Take part of your data or Embody best practice of cloud Move specific service aspects
on demand business applications step- by- computing in unconstrained causing undue cost or
step into non-production DR use greenfield projects management burden
Decouple development and test
environments from operations Understand cloud dynamics and Self contained web projects, Workflows, search indexing,
constraints test during controlled failovers document archiving etc media streaming, document
archiving, constrained databases
Explore elasticity in a sandboxed
environment
Low hanging fruit can be easiest way to ‘cut teeth’
11. Plan evolution & set goals
PoC Production Automation
Understand services Implement monitoring Automate corrective measures
Examples
Test performance Change control and management Auto-scaling
Architect for scale Security management Zero downtime deployments
Build cross functional team capabilities Scalability System backup and recovery
12. Plan evolution & set goals
PoC Production Automation
Understand services Implement monitoring Automate corrective measures
Examples
Test performance Change control and management Auto-scaling
Architect for scale Security management Zero downtime deployments
Build cross functional team capabilities Scalability System backup and recovery
Beanstalk
APIs
Beanstalk Cloud Formation
CLI
Cloud Watch
Auto scaling
IAM
14. Organize your house
Accounts
Create an account structure
that makes sense
Use accounts like environments
where you need separation and
control
e.g
Dev Sandboxes
Test Environments
Business Units
Products & Services
15. Organize your house
Accounts Billing
Create an account structure Control access to billing
that makes sense information
Use accounts like environments Use IAM users to keep billing
where you need separation and information in the master account
control
Consolidate billing into a
e.g single account
Dev Sandboxes Let one account pick up the bill for
Test Environments multiple ‘sub accounts’
Business Units
Products & Services Setup billing alerts and
automated bill reporting
Get CloudWatch notifications when
billing reaches a point and output
csv reports to S3 for analysis
18. Master Account
aws.invoices@mycompany.com
consolidated billing information
Division B
admin@divisionB.com
IAM User2
Dev2
Admin2
19. Master Account
aws.invoices@mycompany.com
consolidated billing information Tags: (key-
value)
Division B e.g Own=Div
Proj=R
admin@divisionB.com
IAM User2
Dev2
Admin2
Tags: Tags: Tags:
Own=Div Own=Div Own=Div
Proj=P Proj=Q Proj=R
20. Master Account
aws.invoices@mycompany.com
consolidated billing information
Operating Co. A Division B Business Unit C
admin@opcoa.com admin@divisionB.com admin@busUnitC.com
User1 User2 User3
IAM
IAM
IAM
Dev1 Dev2 Dev3
Admin1 Admin2 Admin3
Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags:
Own=OpCo Own=OpCo Own=OpCo Own=Div Own=Div Own=Div Own=BusC Own=BusC Own=BusC
Proj=A Proj=B Proj=C Proj=P Proj=Q Proj=R Proj=X Proj=Y Proj=Z
21. Master Account
aws.invoices@mycompany.com
consolidated billing information
Operating Co. A Division B Business Unit C
admin@opcoa.com admin@divisionB.com admin@busUnitC.com
User1 User2 User3
IAM
IAM
IAM
Dev1 Dev2 Dev3
Admin1 Admin2 Admin3
Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags:
Own=OpCo Own=OpCo Own=OpCo Own=Div Own=Div Own=Div Own=BusC Own=BusC Own=BusC
Proj=A Proj=B Proj=C Proj=P Proj=Q Proj=R Proj=X Proj=Y Proj=Z
22. Programmatic billing access
Master Account
aws.invoices@mycompany.com
S3 CSV
consolidated billing information
Operating Co. A Division B Business Unit C
admin@opcoa.com admin@divisionB.com admin@busUnitC.com
User1 User2 User3
IAM
IAM
IAM
Dev1 Dev2 Dev3
Admin1 Admin2 Admin3
Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags:
Own=OpCo Own=OpCo Own=OpCo Own=Div Own=Div Own=Div Own=BusC Own=BusC Own=BusC
Proj=A Proj=B Proj=C Proj=P Proj=Q Proj=R Proj=X Proj=Y Proj=Z
23. Programmatic billing access
Master Account
aws.invoices@mycompany.com
S3 CSV
consolidated billing information
Operating Co. A Division B Business Unit C
admin@opcoa.com admin@divisionB.com admin@busUnitC.com
User1 User2 User3
IAM
IAM
IAM
Dev1 Dev2 Dev3
Admin1 Admin2 Admin3
Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags: Tags:
Own=OpCo Own=OpCo Own=OpCo Own=Div Own=Div Own=Div Own=BusC Own=BusC Own=BusC
Proj=A Proj=B Proj=C Proj=P Proj=Q Proj=R Proj=X Proj=Y Proj=Z
24. Organize your house
Accounts Billing
Create an account structure Control access to billing
that makes sense information
Use accounts like environments Use IAM users to keep billing
where you need separation and information in the master account
control
Consolidate billing into a
e.g single account
Dev Sandboxes Let one account pick up the bill for
Test Environments multiple ‘sub accounts’
Business Units
Products & Services Setup billing alerts and
automated bill reporting
Get CloudWatch notifications when
billing reaches a point and output
csv reports to S3 for analysis
25. Organize your house
Accounts Billing Access Keys
Create an account structure Control access to billing Decide upon a key
that makes sense information management strategy
Use accounts like environments Use IAM users to keep billing Control access to EC2 instances via
where you need separation and information in the master account SSH and embedded public key:
control e.g. EC2 Key Pair per group of
Consolidate billing into a instances, EC2 Key Pair per account
e.g single account
Dev Sandboxes Consider SSH key rotation &
Let one account pick up the bill for
Test Environments multiple ‘sub accounts’
automation
Business Units Limit exposure to private key
Products & Services Setup billing alerts and compromise by rotating keys and
replacing authorized_keys
automated bill reporting listings on running instances
Get CloudWatch notifications when Consider bootstrap automation to
billing reaches a point and output grant developer access with
csv reports to S3 for analysis developer unique keypairs
26. Organize your house
Accounts Billing Access Keys Groups & Roles
Create an account structure Control access to billing Decide upon a key Use IAM Groups to manage
that makes sense information management strategy console users and API access
Use accounts like environments Use IAM users to keep billing Control access to EC2 instances via Provide developers with IAM user
where you need separation and information in the master account SSH and embedded public key: login and unique API access
control e.g. EC2 Key Pair per group of credentials
Consolidate billing into a instances, EC2 Key Pair per account Control & restrict what IAM users
e.g single account can do by placing them in groups
Dev Sandboxes Consider SSH key rotation & with policies
Let one account pick up the bill for
Test Environments multiple ‘sub accounts’
automation
Business Units Limit exposure to private key
Assign EC2 Instances IAM
Products & Services compromise by rotating keys and roles
Setup billing alerts and
replacing authorized_keys Let AWS manage API access
automated bill reporting listings on running instances credentials on running instances by
Get CloudWatch notifications when Consider bootstrap automation to assigning a system entitlement to an
billing reaches a point and output grant developer access with instance
csv reports to S3 for analysis developer unique keypairs e.g instance can only read S3 bucket
27. Identity & access management
Account
Administrators Developers Applications
Jim Brad Reporting
Bob Mark Console
Susan Tomcat
Kevin
28. Identity & access management
Groups Account
Administrators Developers Applications
Jim Brad Reporting
Bob Mark Console
Susan Tomcat
Kevin
Multi-factor authentication
29. Identity & access management
Groups Account Roles
Administrators Developers Applications
Jim Brad Reporting
Bob Mark Console
Susan Tomcat
Kevin
Multi-factor authentication AWS system entitlements
30. IAM policies
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"elasticbeanstalk:*",
Policy driven "ec2:*",
"elasticloadbalancing:*",
Declarative definition of rights for groups "autoscaling:*",
"cloudwatch:*",
Policies control access to AWS APIs "s3:*",
"sns:*"
],
"Resource": "*"
}
]
}
34. Leverage shared security model
Understand your customer & form security stance
Penetration test requests
Your certifications Your processes
External
audience
35. Leverage shared security model
Understand your customer & form security stance
Penetration test requests
Your certifications Your processes
External
audience
IAM
Internal
Administration
audience
Architecture
36. Leverage shared security model
Understand your customer & form security stance
Penetration test requests
Your certifications Your processes
External
audience
IAM AWS Certifications
Internal Regulated
Administration AWS White
audience audience Papers
Architecture AWS QSA Process
37. Leverage shared security model
Understand your customer & form security stance
Engage with security assessors early in adoption cycle
Don’t fear assessment – AWS meets high standards (PCI, ISO27001, SOC2…)
As with any infrastructure provider, security assessments take time
Derive value from architecture reviews early in deployment cycle
38. Leverage shared security model
Understand your customer & form security stance
Engage with security assessors early in adoption cycle
Use comprehensive materials and certifications provided by AWS
http://aws.amazon.com/security/
Risk and compliance paper
AWS security processes paper
CSA consensus assessments
initiative questionnaire
39. Leverage shared security model
Understand your customer & form security stance
Engage with security assessors early in adoption cycle
Use comprehensive materials and certifications provided by AWS
Build upon features of AWS and implement a ‘security by design’ environment
40. Build upon AWS features
Tiered Access Security Groups VPC Direct Connect & VPN
IAM Instance firewalls Subnet control Private connections to VPC
Control users and allow AWS to Firewall control on instances via Create low level networking Secured access to resources in AWS
manage credentials in running Security Groups constraints for resource access, such over software or hardware VPN and
instances for service access as public and private subnets, dedicated network links
(allocation, rotation) CLIs and APIs internet gateways and NATs
Instantly audit your entire AWS
APIs vs Instance infrastructure from scriptable APIs –
Bastion hosts
Provide developer API credentials generate an on-demand IT inventory Only allow access for management
and control access to SSH keys enabled by programmatic nature of of production resources from a
AWS bastion host. Turn off when not
Temporary Credentials needed
Provide developer API credentials
and control access to SSH keys
42. Architect to use cloud strengths
Review application architectures early – assess fit for cloud
? e.g. variable capacity requirements, ‘standard’ technology stacks, reference architectures*
Can cloud benefits be leveraged with minimum effort outlay?
? e.g. Application performance improvement by migration of static content to S3/CloudFront
Will cloud yield cost savings & agility improvements?
? e.g. Faster development cycles for dev/test, reduced cap-ex for application environments
Can automation lead to a more agile & secure service?
? e.g. fully scripted deployments, IAM & EC2 instance roles, rolling deployments
*http://aws.amazon.com/architecture
43. Architect to use cloud strengths
Disposable compute
✓✓ Design systems that can suffer
instance loss
Dispose of compute when it is not
✓ ✓ required
44. Architect to use cloud strengths
Disposable compute
Flexible capacity
✓ ✓ ✓ Design for systems that potentially scale
from zero instances to hundreds
Use Auto-scaling (events, schedules etc) to
✓ ✓ ✓ drive capacity availability
45. Architect to use cloud strengths
Disposable compute
Flexible capacity
✓ ✓ ✓ Utilize 99.999999999% durability of objects
in S3
Scale databases with RDS and use
Cost effective & reliable storage ✓ ✓ ✓ DynamoDB for high throughput NoSQL
46. Architect to use cloud strengths
Disposable compute
Flexible capacity
✓ ✓ ✓ Automate everything from scaling to
instance recovery from failure
Cost effective storage
Automation and control
47. Bootstrapping – custom AMIs
Instanc
e
AMI
Custom machine
1 Create instance for your OS choice image
2 Configure environment
Auto-scaling
Manual deployments
3 Install software Programmatic deployments
4 Create AMI from instance
5 Launch fully configured instances from AMI
48. Bootstrapping – metadata service
Instanc
e
Metadata service contains wealth of information about an instance AMI
http://169.254.169.254/latest/meta-data Custom or standard
machine image
ami-id local-hostname Receive custom
Metadata
data to drive
ami-launch-index local-ipv4 Service
bootstrapping
ami-manifest-path mac
block-device-mapping network
hostname placement
instance-action profile
instance-id public-hostname
Instance-type public-ipv4
kernel-id public-keys
reservation-id
49. Bootstrapping – metadata service
Instanc
e
Metadata service contains wealth of information about an instance AMI
http://169.254.169.254/latest/meta-data Custom or standard
machine image
+ user data Receive custom
data to drive
Metadata
Service
bootstrapping
Scripts in user-data field of metadata will be executed on launch
e.g.
#!/bin/sh
yum -y install httpd
chkconfig httpd on
/etc/init.d/httpd start
Or:
<powershell>
…
</powershell>
50. Bootstrapping – metadata service
Instanc
e
Metadata service contains wealth of information about an instance AMI
http://169.254.169.254/latest/meta-data Custom or standard
machine image
+ user data Receive custom
data to drive
Metadata
Service
bootstrapping
Scripts in user-data field of metadata will be executed on launch
Install software e.g. web server, app server, proxy
Pull data and application packages from S3
Publish metadata for instance to other systems e.g. monitoring systems
Setup security profile of instance based upon intended use e.g. pull latest config
56. Architect to use cloud strengths
Elastic Load Balancing Route 53 RDS Auto-scaling
Use at regional level Leverage SLA Scale databases without Dynamically scale resources &
Combined with autoscaling will Improve application reliability with admin overhead control costs
balance requests and resource Route 53’s SLA on requests served Choose instance size for databases Only provision the resources that
capacity across availability zones and scale up over time are required with scale up and cool
Weighted routing down policies that match demand
Within VPC Perform A/B analysis, and staged Add high availability from
Use to loadbalance between application roll-outs by moving a management console
application tiers within an portion of traffic to new Create master-slave configurations
availability zone infrastructure and read-replicas. AWS takes care of
the failover and recreation of a new
Instance migrations Control TTLs and updates slave in event of master DB loss
Easily move instances from dev Take absolute control of DNS
environments to test environments updates for more decisive system
by moving between ELBs updates
58. Services not software
30% 70%
Self Managed
Software & Your Managing All of the
Infrastructure Business “Undifferentiated Heavy Lifting”
AWS
More Time to Focus on Configuring Your
Cloud-Based
Your Business Cloud Assets
Infrastructure & Services
70% 30%
59. Services not software
Relational Database Service
Use RDS for databases Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
DynamoDB Use DynamoDB for
Provisioned throughput NoSQL database high performance key-
Fast, predictable performance
value DB
Fully distributed, fault tolerant architecture
60. Services not software
Processing results
Amazon SQS Reliable message
Reliable, highly scalable, queue service
Amazon SQS
queuing without
for storing messages as they travel
between instances
additional software
Processing
task/processing
trigger 1
2
Push inter-process Simple Workflow Task A
workflows into the Reliably coordinate processing steps
Task B 3
across applications
cloud with SWF (Auto-scaling)
Integrate AWS and non-AWS resources
Manage distributed state in complex
systems Task C
61. Services not software
Document
Cloud Search Server
Don’t install search
Elastic search engine based upon
software, use
Amazon A9 search engine
CloudSearch Fully managed service with sophisticated
feature set
Search
Scales automatically
Server
Results
Elastic MapReduce
Elastic Hadoop cluster
Process large volumes
Integrates with S3 & DynamoDB of data cost effectively
Leverage Hive & Pig analytics scripts with EMR
Integrates with instance types such as
spot
63. Be elastic and cost optimized
Elastic Load Balancing Auto-scaling policies
Scalability
Cost Optimization
Availability
Instance types and sizes
64. Auto-scaling policies
Manually By Schedule
Send an API call or use CLI to Scale up/down based on date
launch/terminate instances – and time
Only need to specify capacity
change (+/-)
By Policy Auto-Rebalance
Scale in response to changing Instances are automatically
conditions, based on user launched/terminated to
configured real-time ensure the application is
monitoring and alerts balanced across multiple Azs
65. Auto-scaling policies
Manually By Schedule
Preemptive manual scaling
Send an API call or use CLI to Regular scaling up and down
Scale up/down based on date
of capacity
launch/terminate instances – ofand time
instances
Only need marketing event add 10
e.g. before a to specify capacity e.g. scale from 0 to 2 to process SQS
more instances messages every night or double
change (+/-) capacity on a Friday night
By Policy Auto-Rebalance
Scale in response to changing Instances are automatically
Dynamic scale based upon
conditions, based on user Maintain capacity across
launched/terminated to
configuredmetrics
custom real-time availability zones
ensure the application is
e.g. SQS queue depth, Average CPU e.g. Instance availability maintained in
monitoring and alerts
load, ELB latency
balanced across multiple Azs
event of AZ becoming unavailable
66. Instance types
On-demand instances Reserved instances Spot instances
Unix/Linux instances start at 1- or 3-year terms Bid on unused EC2 capacity
$0.02/hour
Pay low up-front fee, receive significant hourly Spot Price based on supply/demand,
Pay as you go for compute power discount determined automatically
Low cost and flexibility Low Cost / Predictability Cost / Large Scale, dynamic workload handling
Pay only for what you use, no up-front Helps ensure compute capacity is available
commitments or long-term contracts when needed
Use Cases:
Use Cases:
Use Cases: Applications with flexible start and end times
Applications with short term, spiky, or
unpredictable workloads; Applications with steady state or predictable Applications only feasible at very low compute
usage prices
Application development or testing
Applications that require reserved capacity,
including disaster recovery
68. Everything is programmable
Access everything Achieve the highest levels
via CLI, API or Compute of automation
Console Security Scaling sophistication with ease
CDN Backup
DNS Database
Storage Load Balancing
Workflow Monitoring
Networking
Messaging
69. Elastic Beanstalk OpsWorks CloudFormation
Quickly deploy and manage apps in AWS…
70. Elastic Beanstalk OpsWorks CloudFormation
CloudFormation components & terminology
CloudFormation
Template Stack
JSON formatted file Framework Configured AWS services
Parameter definition Stack creation Comprehensive service support
Resource creation Stack updates Service event aware
Configuration actions Error detection and rollback Customisable
71. Elastic Beanstalk OpsWorks CloudFormation
Powerful management framework with Chef support
Stack Layers Apps Management
Managed Collection of Your application Management
environment resources assets services
Definition of environment Blueprint for a Resources to deploy Scaling, cloning, user
such as production or test collection of resources and run in layers access, self healing
(instances, EBS, EIPs
etc)
73. Offering
Basic 24x7x365 ✓
Developer Forum Access ✓
Documentation ✓
Business Access to support Support for
HealthChecks
Enterprise
74. Offering
Basic 24x7x365 ✓
Developer Forum Access ✓
Documentation ✓
Business Access to support Support for
HealthChecks
Enterprise
75. Basic Offering
24x7x365 ✓
Developer Forum Access ✓
Documentation ✓
Business Access to support Email
Named Contacts 1
Enterprise
Fastest Response Time 12 Hours
Architecture Support Building Blocks
Best Practice ✓
Diagnostics Tools ✓
76. Basic Offering
24x7x365 ✓
Developer Forum Access ✓
Documentation ✓
Business Access to support Phone, Chat,
Email
Enterprise Named Contacts 5
Fastest Response Time 1 Hour
Architecture Support Use Case
Guidance
Best Practice ✓
Diagnostics Tools ✓
Direct Routing ✓
3rd Party Software ✓
Trusted Advisor ✓
77. Basic Offering
24x7x365 ✓
Developer Forum Access ✓
Business Documentation ✓
Access to support Phone, Chat,
Email
Enterprise Named Contacts Unlimited
Fastest Response Time 15 Minutes
Architecture Support Application
Architecture
Best Practice ✓
Diagnostics Tools ✓
Direct Routing ✓
3rd Party Software ✓
Trusted Advisor ✓
Direct TAM Access ✓
White Glove Case Handling ✓
Management Business Review ✓
79. Business and Enterprise Support has been enhanced to include best
practice audits via AWS Trusted Advisor
Security Fault Tolerance Cost Optimization
Open ports in Security Groups EBS snapshot age Unused Elastic Ips
World access (/0 CIDR) ELB Optimization Underutilized EC2 instances
IAM use Availability Zones
81. 3rd Party Software Support Enhancements
Operating Systems 3rd Party Software
Operating Systems including: Common application stack components
including:
Amazon Linux
Apache and IIS web servers
Ubuntu Amazon SDKs
Red Hat Enterprise Linux Sendmail
SUSE Linux Postfix
FTP
Microsoft Windows 2003 & 2008 R2
Disk Management tools (LVM, RAID) VPN
Solutions (OpenVPN, RRAS)
Databases (MySQL, SQL Server)
83. Choose your use case well
Organize your environments
Think security
Architect to cloud strengths
Services not software
Be elastic & cost optimized
Use frameworks where appropriate
Get supported