Running out of capacity on your NAS? Tired of buying and maintaining storage systems for file shares, media archives, or high-performance shared file systems? Learn how you can use AWS storage services to help eliminate the capital expense and operational complexity of on-premises file storage. We provide guidance how to use AWS’s in-cloud file storage service Amazon EFS, as well as how to connect on-premises file workloads to data stored in Amazon S3 via the AWS Storage Gateway. Hear examples from customers such as Celgene Corporation, who are using these services in hybrid and in-cloud architectures, to take advantage of AWS durability, performance, and economics.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:Invent
Files in AWS : Over coming S tor ag e Challeng es
for Common File Use Cases with S pecial Guest
Celg ene
P a u l R e e d , S e n i o r P r o d u c t M a n a g e r , A W S
P r a s h a n t h B u n g a l e , S e n i o r P r o d u c t M a n a g e r , A W S
L a n c e S m i t h , I T D i r e c t o r , C e l g e n e C o r p o r a t i o n
S T G 3 1 0
N o v e m b e r 2 9 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data movement
OnlineOffline
Data security
and management
Complete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow Family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
Amazon CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
Amazon QuickSight

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File storage is the most common storage
type for application developers
Interface: appears as a native file system on a client accessing it, OS API,
accessible from multiple clients concurrently
Semantics: strong data consistency, locking, directory structure
Permissions model: standard OS model for specifying read/write/execute
permissions for users logged in to the OS

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operating shared file storage today is a pain
Application
owner or
developer
IT administrator
• Estimate demand
• Procure hardware
• Set aside physical space
• Set up and maintain hardware (and network)
• Manage access and security
• Provide demand forecasts/business case
• Add lead times and extra coordination to your schedule
• Limit your flexibility and agility

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk file storage
• Hybrid use cases
• Windows file services with AWS Volume Gateway
• Content storage and distribution with File
Gateway
• File storage for in-cloud workloads with Amazon EFS
• EFS File Sync
• Celgene Corp.

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Delivering Windows File Services
on-premises, tiered & protected on AWS

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What you’ve
had to
manage
V
M V
M V
M V
M V
M V
M V
M V
M V
M
V
M V
M V
M V
M V
M V
M
V
M V
M V
MV
M
V
M
Delivering Windows File Services
What
they are
used to
Storage
Gateway
Service
But how do I get AWS
storage for Windows
on-premises?
An
alternative
And what do I
need on-
premises now?
V
M
Users

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
O n - p r e m i s e s v o l u m e s b a c k e d b y A m a z o n S 3 w i t h A m a z o n E B S
s n a p s h o t s
Volume Gateway
Application
Server
Volume Stored in
Amazon S3
iSCSI
Customer Premises
• Block storage in Amazon S3 accessed
via on-premises gateway appliance VM
• Compresses data in transit and at rest
• Up to 1 PB of total volume storage per
gateway appliance
• Create EBS snapshots of on-premises
volumes
• Create volumes from EBS snapshots
• Two modes: volume stored and volume
cached
Storage
Gateway

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Datacenter Windows Server + Storage Gateway
& cloud backend
S M B h o s t e d o n s i t e , b l o c k s s t o r e d d u r a b l y i n A m a z o n S 3
Private datacenter
us-west-2
Storage
Gateway
VM
Storage
Gateway
Volume
Windows
Clients
Windows
Server
HTTPSiSCSISMB
Amazon
EBS
snapshots
EC2
instance
Win
Server
EBS
Volume

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
High availability Windows Server storage
L o c a l H A wi t h r e m o t e c l o u d D R
us-west-1
Data center
Win
Clients Storage
Gateway
VMs
Microsoft
DFS
HTTPSiSCSI
HTTPSiSCSI
SMB
Storage Gateway
Service
us-east-1
Microsoft
DFS
HTTPSiSCSI
Storage Gateway
Service
us-east-1
us-west-1

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid Cloud Content Storage & Distribution
with File Gateway & Amazon S3

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Store, protect, and scale
cost-effectively
Global online access
and distribution
Process, transform, and
extract value
What do (we hear) you want to do?

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
O n - p r e m i s e s f i l e s t o r a g e m a i n t a i n e d a s o b j e c t s i n A m a z o n S 3
• Storage in your S3 buckets with 1:1 mapping from files-to-objects
• File metadata stored in object metadata
• Full control over how your data is accessed and stored
Customer Premises
Glacier
HTTPS
NFS
v3 / v4.1
Application
Server
Objects in your
S3 bucket
Storage
Gateway VM

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
s3://{bucket name}/{folder1}/{file1}
s3://{bucket name}/{folder1}/{file2}
s3://{bucket name}/{folder1}/{file3}
s3://{bucket name}/{folder1}/{file4}
s3://{bucket name}/{folder2}/{file1}
s3://{bucket name}/{folder2}/{file2}
File Gateway
O n e - t o - o n e m a p p i n g f r o m f i l e s - t o - o b j e c t s
nfs://{IP}/{share name}/{folder1}/{file1}
nfs://{IP}/{share name}/{folder1}/{file2}
nfs://{IP}/{share name}/{folder1}/{file3}
nfs://{IP}/{share name}/{folder1}/{file4}
nfs://{IP}/{share name}/{folder2}/{file1}
nfs://{IP}/{share name}/{folder2}/{file2}
File Gateway
VM
NFS client Objects in your
S3 bucket

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
C o n t r o l h o w y o u r d a t a i s a c c e s s e d a n d s t o r e d
NFS/file options export
• IAM role for access
• Storage class
• Object encryption with KMS
• Guess MIME type ** NEW! **
S3 options per bucket
• List of allowed clients
• Read-only/read-write exports
• User squashing
• Default object ownership (uid/gid)
• Default object permissions
File Gateway
VM
NFS client Objects in your
S3 bucket

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
M e t a d a t a a n d w o r k l o a d a u t o m a t i o n
Metadata
and object
inventory
Data
cache
File
share
File Gateway VM
RefreshCache
API
Amazon
CloudWatch
Events
NFS client Objects in your
S3 bucket
ls /foo/*

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Miami regional office
New York regional office
Los Angeles headquarters
Content distribution with File Gateway
Amazon S3
bucket
NFS client
(writer)
File Gateway
VM
File Gateway
VM
NFS client
(read-only)
NFS client
(read-only)
File Gateway
VM
Request
upload
notification
1
Amazon
CloudWatch
2
3
Client writes files and calls
NotifyWhenUploaded API…
1 When files are uploaded, the gateway
sends a CloudWatch Event…
2 Event triggers RefreshCache API on
gateways in the remote offices
3

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multi-region file distribution & access
Storage
Gateway VM
S3 BucketCross region
replication
Los Angeles HQ
NFS
client
S3 Bucket
US-West – Oregon region
Melbourne site
Sydney, Australia region
Read-only
NFS client
Storage
Gateway VM
RefreshCache
API

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer data center
Hybrid Processing: File Gateway, Objects in S3
& AWS services
Amazon S3
bucket
NFS
client
File
Gateway
VM
Refresh
Cache
Amazon
CloudWatch
Amazon
Athena
Amazon
QuickSight
Amazon
EMR
AWS
Lambda
Amazon
Rekognition

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer data center
Storage offload with AWS Snowball & File
Gateway
Amazon S3
bucket
NFS client
File Gateway
VM
RefreshCache
AWS Snowball

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File storage for in-cloud workloads
with Amazon Elastic File Service
(Amazon EFS)

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Before EFS…DIY file storageAZ-a
Clients
Storage
Volumes
AZ-b
Storage
Volumes
File Server
File Server

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key features of Amazon EFS
ScalableSimple Elastic
Highly available and durable

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Designed for a wide spectrum of needs
Scale-out jobs Metadata-intensive jobs
Analytics
M&E workflows
Web serving
Content Management
Database backups
Enterprise apps
& messaging
Dev tooling
Home directories
Container storage
Low latency and serial I/OHigh throughput and parallel I/O

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EFS customers
Web serving + Content management
Media and entertainment workflowsDatabase backups
Big data and analytics
Enterprise applications
Developer tools Home directories Container storage

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security model and security features
Control
network traffic
using VPC security
groups and network
ACLs
Control file and
directory access
using POSIX
permissions
Control administrative
access (API access)
using AWS Identity and
Access Management (IAM)
(action-level and
resource-level
permissions)
Encrypt
data at rest
using keys
managed in AWS
KMS

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EFS architecture
NFS Clients Mount Target
Mount Target
Mount Target
AZ-aAZ-bAZ-c
DNSName
Amazon EFS
File System
NFS Clients
NFS Clients

28. How to access a file system from an instance
• You “mount” a file system on an Amazon EC2 instance (standard
command)—the file system appears like a local set of directories
and files
• An NFS v4.1 client is standard on Linux distributions
mount –t nfs4 –o nfsvers=4.1
[file system DNS name]:/
/[user’s target directory]

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choose the performance mode
best suited to your workload
Mode What it is for Advantages Tradeoffs When to use
General
purpose
(default)
Latency-
sensitive
applications and
general-purpose
workloads
Lowest latencies
for file operations
Limit of 7k
ops/sec
Best choice for
most workloads
Max I/O
Large-scale and
data-heavy
applications
Virtually
unlimited ability
to scale out
throughput/IOPS
Slightly
higher
latencies
Consider for
large scale-out
workloads

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EFS distributed data storage design
File systems distributed across
unconstrained number of servers
• Avoids bottlenecks/constraints
of traditional file servers
• Enables high levels of
aggregate IOPS/throughput
Data also distributed
across Availability Zones
(durability, availability)
EC2
EC2
…
EC2
EC2
…
EC2
EC2
…

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
An implication of per-operation
latency: I/O size impacts throughput
of serialized operations
4 KB 32 KB 256 KB 2 MB 16 MB
I/O size
Throughput

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to take advantage of EFS distributed
architecture: Parallelize
0
5000
10000
15000
20000
25000
30000
0 20 40 60 80 100 120 140 160
IOPS
# of Total Threads

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EFS economics
$0.30/GB-Month (US Regions)
$0.33/GB-Month (EU Ireland)
$0.36/GB-Month (EU Frankfurt)
$0.36/GB-Month (AP Sydney)
No minimum commitments
or up-front fees
No need to provision
storage in advance
No other fees, charges,
or billing dimensions

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Before EFS… Costs of DIY file storageAZ-a
Clients
Storage
volumes
AZ-b
Storage
volumes
File server
File server
EC2 instance
costs
Inter-AZ data
transfer costs
EBS volume
costs

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TCO example: For storing 500 GB,
EFS is 70% less than DIY
EFS cost: (500GB * $0.30/GB-month) = $150 per month
For DIY, you might provision 600 GB of EBS (i.e., ~85% utilization):
Storage (2x 600 GB EBS gp2 volumes): $120 per month
Compute (2x m4.xlarge instances): $290 per month
Inter-AZ data transfer costs (est.): $130 per month
Total $540 per month

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EFS best practices
• Test your application using actual workflows
• Use General Purpose mode for lowest latency, Max-I/O for scale-out
• Use Linux kernel version 4.0 or newer, mount via NFSv4.1
• To optimize, look for opportunities to:
• Aggregate I/O
• Perform async operations
• Parallelize
• Cache
• Check your burst credit earn/spend rate when testing—ensure
sufficient amount of storage

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EFS file systemExisting file
system
Corporate data center AWS region
Getting data into EFS
Problem: customers want to run file-based applications on EC2, but lack
tools to efficiently move existing on-premises data into EFS

38. EFS file system
AWS region
But it’s not as simple as it might seem at first...
Existing file system
Corporate data center
Packet loss
Firewall
Network
Latency
Unsecure
Scripts
Monitoring

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introducing Amazon EFS File Sync
Fast
Up to 5x faster than
standard Linux copy tools
Sync data from existing file systems into EFS file systems
Secure
Encrypted parallel data
transfer to AWS
Simple
Set up and manage easily
from the AWS Console
File systems from on-premises to EFS
DIY in-cloud file systems to EFS
EFS file systems between AWS regions
Use EFS File Sync to copy…

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Celgene uses AWS file storage for
high-performance & hybrid IT needs
L a n c e S m i t h , D i r e c t o r o f I T

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Celgene, our AWS journey and strategy
• Cloud migration with Amazon EFS
• Hybrid cloud with AWS Storage Gateway

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Celgene is building a preeminent global
biopharmaceutical company focused on
the discovery, development, and
commercialization of innovative
therapies for patients with cancer,
immune-inflammatory, and other
unmet medical needs.

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Celgene’s AWS Journey & Strategy
2014
2015
2016
2017
2018+
R&D-led | Reduce heavy lifting | SaaS & COTS | Cloud-first IT | Cloud future
WAF / SQS
Direct Connect
CloudFront
IoT
Storage Gateway
AWS Lambda
S3
EC2
Amazon
RDS
Aurora
EFS
EMR
WorkSpaces

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Genomics HPC
Accelerating Innovation
Molecular chemistry Collaboration: hospitals
and universities
Corporate IT Legacy applications

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Data protection—integrity
• Intensity of maintenance reduction
• Performance, but not at any price
• TCO, not just OpEx cost
• Consistency of research and business
processes
• Design for failure
• Separate compute from storage
• Reduce manual labor everywhere
• Don’t DIY if you can avoid it
• Re-platform without re-factoring
O u r b e s t p r a c t i c e sW h a t w e c a r e a b o u t
Legacy application goals—storage

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What we do with EFS?
• Moved HPC/life-science applications
• Traditional block storage use cases
• Performance NFS
• Enabled us to move more into AWS
Why EFS for us?
• Simple, durable, agile
• No maintenance, no patches, no
performance tweaking
• No re-engineering the applications
Why not something else?
• Friends don’t let friends build DIY
parallel File Systems
• We love S3, but many applications
aren’t object-aware
EFS—What for & why?
A m a z o n
E l a s t i c F i l e S y s t e m

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HPC—Computational chemistry
Before
On-premises NAS/GPFS
Alternative
App. vendor’s cloud
Parallel FS on EC2
Now
EC2 + EFS + DX +…
Tip
Phone a friend (@AWS)
VPC subnet
Auto
Scaling
Amazon
CloudWatch
AWS
CloudTrail
EFS
VPC subnet
Auto
Scaling
Auto
Scaling
Celgene Sites
EC2 Clients EC2 Clients
EC2 Clients

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HPC—Genomic
VPC subnet
VPC subnet
Auto
Scaling
CloudWatch
SQS
EFS
bastion
S3
VPCe
S3Before
On-premises projects =
TBs
NFS on EC2…
Now
Object + file combo
Migrating from bastion on
EC2 to workspaces
Projects = PBs
Benefit
Scalability changes
business processes,
innovation, and outcomes
EC2 Clients

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
mSaaS collaboration architecture
Plugin
Pipeline
Data
processing
Extraction
processing
Transaction
processing
Web servers
Amazon
SQS
Amazon
S3
Logging SSO
Platform
Services
RDS
WAF
users
EFS

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Connectivity
• Multi-account and
VPC model
• AWS Direct Connect
+ private circuits
AWS Local Region
Customer Router
AWS Direct
Connect Location
AWS Cage
Customer Cage
AWS Direct
Connect Router
CA EUNJ
Private Virtual Interface
Public Virtual Interface
Customer Circuit
Network Legend
us-west-1
us-east-1
eu-central-1
Celgene Location
AWS Region
Equinix Colocation
Map Legend

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use cases Challenges Previous options
Hybrid cloud file storage
• Impossible to change
applications in near-term
• Very hard to change people
and processes
• Data transfers and latency
• Offline data movement
possible, but lengthy
• S3 upload clients, but
resistance to change by users
• Multiple backup tools
• One backup software cost
$25K for one instrument; we
have 1K instruments…
• Rein in ”hidden” files
stored on local lab disk
• System archives, files
• Data migrations

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid workloads
• Archive
• Instrumentation
• Hybrid compute
S3 Bucket
EC2-based
applications
RDS
Databases
Lab
Instrument
PCs
Laboratory
Instrumentation
Experiments
EFS File System
AWS File
Gateway
On-premises
Servers
3D Rendering

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Networking for hybrid cloud
Corporate data center
Corporate data center
User subnet
Server subnet
Research
Compute
subnet
Laboratory
Compute
subnet
Router
AWS cloud
S3 Bucket
EFS
AWS File
Gateway
EFS EC2 Clients
Mount Target
EC2 Clients
Mount Target

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• EFS is easy
• EFS is scalable and performance storage
• Storage Gateway enables hybrid cloud and eases
enterprise migration into AWS
AWS Storage Team
AWS Account Team
Celgene HPC
T h a n k s !S t o r a g e m i c r o s e r v i c e s
Summary

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more in these sessions
Tomorrow
• STG204 – Migrating Large Scale Data Sets to the Cloud – Thur. @3:15pm
• STG307-R: Deep Dive on Amazon Elastic File System – Thur. @12:15pm
Catch on YouTube
• STG201 – Storage State of the Union
• STG203 – Get rid of tape & modernize backup w/ AWS, featuring Southern Oregon University
• ENT322 – Disaster Recovery with AWS: Tiered Approaches
• STG320 – Workshop: Hybrid Storage Unleashed, featuring Storage Gateway

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital
training courses including
advanced training on storage
• Deep Dives on S3, EFS, and EBS
• Migrating and Tiering Storage
to AWS (Hybrid Solutions)
At re:Invent
• Visit hands-on labs at the
Venetian
• Attend a proctored
“Introduction to EFS” Spotlight
Lab on Thursday at 3 p.m. at
the Venetian
• Meet storage experts at the Ask
the Experts in hands-on labs
room at the Venetian
New storage training

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions?
Thank you!