Amazon launched Amazon Web Services in 2006, which has since become a mainstream cloud computing platform. The document discusses AWS services for research computing, including tools to quickly deploy resources, securely store and analyze large datasets, and control costs through spot instances and budget management. It also highlights several scientific organizations successfully using AWS for research.

1. Amazon Web Services
Amazon

2. AWS Research Cloud Program & Researcher’s Handbook
Worldwide Research & Technical Computing
2017-01-13

3. “… the online book and decorative pillow seller Amazon.com
swooped in and, in 2006, launched its own computer rental system—
the future Amazon Web Services. The once-fledgling service has
since turned cloud computing into a mainstream phenomenon …”
Source: Bloomberg Business - April 22, 2015

5. missing manual
Written by Amazon’s Research Computing
community for scientists.
• Explains foundational concepts about how AWS
can accelerate time-to-science in the cloud.
• Step-by-step best practices for securing your
environment to ensure your research data is safe
and your privacy is protected.
• Tools for budget management that will help you
control your spending and limit costs (and
preventing any over-runs).
• Catalogue of scientific solutions from partners
chosen for their outstanding work with scientists.
aws.amazon.com/rcp

6. Disrupting science, wherever it’s happening.

7. IT’S
ABOUT
SCIENCE,
NOT
SERVERS. #AWSresearchcloud
aws.amazon.com/rcp

8. Prediction
Experiment
Results
Refine
Model
Hypothesis
Credit: Aristotle

10. data point

11. DATASETS,
TOOLS &
TECHNIQUES
#AWSresearchcloud
aws.amazon.com/rcp

13. laptop server
server cluster
CPU GPU
… in minutes.

14. We’ve been using Lambda, a
brand new AWS service launched
this year, which helps our
software to quickly adjust
compute resources to match the
complexity of the analysis task. It
processes billions of base pairs in
its off-target search by
subdividing the job into
independent, modular tasks that
can be run in parallel.
A typical GT-Scan2 job takes less
than a minute and thanks to
Lambda we can keep the runtime
constant irrespective of how
complex the task.

15. Cray Supercomputer

17. spare capacity
actual
Peak: 58K cores
Valley: 12K cores

18. THERE’S
NO
QUEUE IN
THE
CLOUD. #AWSresearchcloud
aws.amazon.com/rcp

19.  750+ popular scientific applications
AWS Marketplace
immediately
Introducing Alces Flight - self-scaling HPC clusters instantly ready to compute, billed by the hour and using
the AWS Spot market by default to achieve supercomputing for ~1c per core per hour.
http://boofla.io/u/alcesFlight

20. INSTANT

21. Wall clock time: ~1 hour Wall clock time: ~1 week
Cost: the same

22. AWS Region
Availability Zone
regions are sovereign your data never
leaves
Americas
• AWS GovCloud (2)
• US West
• Oregon (3)
• Northern California (3)
• Northern Virginia (5)
• Ohio (3)
• Montreal (2)
• São Paulo (3)
Europe
• Ireland (3)
• Frankfurt (2)
• London (2)
• Paris
Asia Pacific
• Singapore (2)
• Sydney (3)
• Tokyo (3)
• Seoul (2)
• Mumbai (2)
• Beijing (2)
• Ningxia

23. Account Support
Support
Managed
Services
Professional
Services
Partner
Ecosystem
Training &
Certification
Solution
Architects
Account
Management
Security & Pricing
Reports
Technical Acct.
Management
Marketplace
Business
Applications
DevOps Tools
Business
Intelligence
Security
Networking
Database &
Storage
SaaS
Subscriptions
Operating
Systems
Mobile
Build, Test,
Monitor Apps
Push
Notifications
Build, Deploy,
Manage APIs
Device Testing
Identity
Enterprise
Applications
Document
Sharing
Email &
Calendaring
Hosted
Desktops
Application
Streaming
Backup
Game
Development
3D Game
Engine
Multi-player
Backends
Mgmt. Tools
Monitoring
Auditing
Service Catalog
Server
Management
Configuration
Tracking
Optimization
Resource
Templates
Automation
Analytics
Query Large
Data Sets
Elasticsearch
Business
Analytics
Hadoop/Spark
Real-time Data
Streaming
Orchestration
Workflows
Managed
Search
Managed ETL
Artificial
Intelligence
Voice & Text
Chatbots
Machine
Learning
Text-to-Speech
Image Analysis
IoT
Rules Engine
Local Compute
and Sync
Device
Shadows
Device
Gateway
Registry
Hybrid
Devices & Edge
Systems
Data Integration
Integrated
Networking
Resource
Management
VMware on
AWS
Identity
Federation
Migration
Application
Discovery
Application
Migration
Database
Migration
Server
Migration
Data Migration
Infrastructure Regions
Availability
Zones
Points of
Presence
Compute Containers
Event-driven
Computing
Virtual
Machines
Simple Servers Auto Scaling Batch
Web
Applications
Storage Object Storage Archive Block Storage
Managed File
Storage
Exabyte-scale
Data Transport
Database MariaDB
Data
Warehousing
NoSQLAurora MySQL Oracle SQL ServerPostgreSQL
Application
Services
Transcoding Step Functions Messaging
Security
Certificate
Management
Web App.
Firewall
Identity &
Access
Key Storage &
Management
DDoS
Protection
Application
Analysis
Active Directory
Dev Tools
Private Git
Repositories
Continuous
Delivery
Build, Test, and
Debug
Deployment
Networking
Isolated
Resources
Dedicated
Connections
Load Balancing Scalable DNSGlobal CDN
The
AWS
Platform

24. EC2
http://aws.amazon.com/ec2/instance-types/
There’s several dozen
EC2 compute instances
to choose from, each
family is optimized for
different things.
One size does not fit
all.
• GPUs
• FPGAs
• Skylake

25. #!/bin/bash
instance_id=$(aws ec2 run-instances --image-id ami-12345
--query Instances[].InstanceId
--output text)
instance_state=$(aws ec2 describe-instances --instance-ids $instance_id
--query 'Reservations[].Instances[].State.Name')
while [ "$instance_state" != "running" ]
do
sleep 1
instance_state=$(aws ec2 describe-instances --instance-ids $instance_id
--query 'Reservations[].Instances[].State.Name')
done

27. “The Zooniverse is heavily reliant on Amazon Web
Services (AWS), particularly Elastic Compute
Cloud (EC2) virtual private servers and Simple
Storage Service (S3) data storage. AWS is the
most cost-effective solution for the dynamic needs
of Zooniverse’s infrastructure …”
http://wwwconference.org/proceedings/www2014/companion/p1049.pdf
The World’s Largest Citizen Science Platform
… cost is a factor – running a central API means that when the Zooniverse is quiet and
there aren’t many people about we can scale back the number of servers we’re running
(automagically on Amazon Web Services) to a minimal level.

28. missing manual
Written by Amazon’s Research Computing
community for scientists.
• Explains foundational concepts about how AWS
can accelerate time-to-science in the cloud.
• Step-by-step best practices for securing your
environment to ensure your research data is safe
and your privacy is protected.
• Tools for budget management that will help you
control your spending and limit costs (and
preventing any over-runs).
• Catalogue of scientific solutions from partners
chosen for their outstanding work with scientists.
aws.amazon.com/rcp

30. Solving Procurement Challenges
Invoice-backed billing
means no need for credit
cards in order to sign up to
AWS and use services.
Simple procedure
Global Data Egress Waiver
Single Sign-up
aws.amazon.com/rcp

31. Cost Control & Budgeting
Cost Explorer
AWS Budgets
Simple, safe &
secure.
aws.amazon.com/rcp

32. Introducing Alces Flight - Self-scaling HPC clusters instantly
ready to compute, billed by the hour and set to achieve
supercomputing for ~1c per core per hour.
1,150+ scientific applications pre-installed and ready to run in
AWS Marketplace (the cloud’s “Application Store”) and launched
within minutes.
Most journals & funding bodies are mandating that data and
methods be shared in an open way to ensure repeatability or
falsifiability. Figshare enables researchers to easily adhere to
these principles by making research outputs shareable and
discoverable.
aws.amazon.com/rcp

33. missing manual
Written by Amazon’s Research Computing
community for scientists.
• Explains foundational concepts about how AWS
can accelerate time-to-science in the cloud.
• Step-by-step best practices for securing your
environment to ensure your research data is safe
and your privacy is protected.
• Tools for budget management that will help you
control your spending and limit costs (and
preventing any over-runs).
• Catalogue of scientific solutions from partners
chosen for their outstanding work with scientists.
aws.amazon.com/rcp

34. CFD Direct
aws.amazon.com/rcp