1. Data-driven innovation
Dr. Matt Wood
matthew@amazon.com
@mza

2. Hello

3. Hello

4. Data

5. DNA

6. Chromosome 11 : ACTN3 : rs1815739

7. Chromosome X : rs6625163

8. Chromosome 19 : FUT2 : rs601338

9. Chromosome 2 : rs10427255

10. Chromosome 10 : rs7903146
TYPE II

11. Chromosome 15 : rs2472297
+0.25

12. I know this, because...

14. A
T
C
G
G
T
C
C
A
G
G

15. A
T A
C G
G C
G C
Transcription
T A
C G
C G
A U
G C
G C

16. A
T A
C G Ser
G C
G C
Transcription Translation
T A Glu
C G
C G
A U Val
G C
G C

19. Chromosome 11 : ACTN3 : rs1815739

20. Chromosome X : rs6625163

21. Chromosome 19 : FUT2 : rs601338

22. Chromosome 2 : rs10427255

23. Chromosome 10 : rs7903146
TYPE II

24. Chromosome 15 : rs2472297
+0.25

25. I know all that, because...

26. Human Genome Project

27. 40 species
ensembl.org

28. Compare

29. Change

30. Less

33. Compare

34. Transformative

36. Data generation costs are falling
everywhere

37. Customer segmentation,
financial modeling,
system analysis,
line of sight,
business intelligence.

38. Opportunity

39. Transformation

40. Innovation

41. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

42. lower cost,
increased throughput
Generation
Collection & storage
Analytics & computation
Collaboration & sharing

43. lower cost,
increased throughput
Generation
highly constrained
Collection & storage
Analytics & computation
Collaboration & sharing

44. Barrier

45. Data generation
X
challenge

46. Analytics
challenge

47. Accessibility
challenge

48. Enter the AWS Cloud

49. Utility

50. Remove constraints

51. Data-driven innovation

52. Distributed

53. 2

54. Software for distributed
storage & analysis
2

55. Software for distributed
storage & analysis
2
Infrastructure for distributed
storage & analysis

56. Software
Frameworks for
data-intensive work loads.
Distributed by design.

57. Infrastructure
Platform for
data-intensive work loads.
Distributed by design.

58. Support the data timeline

59. Generation
highly constrained
Collection & storage
Analytics & computation
Collaboration & sharing

60. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

61. Lower the barrier to entry

62. Agility

63. Responsive

64. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

65. Generation
DynamoDB
Analytics & computation
Collaboration & sharing

66. Generation
DynamoDB
EC2, Elastic MapReduce
Collaboration & sharing

67. Generation
DynamoDB
EC2, Elastic MapReduce
S3, Public Datasets

68. Tools and techniques for working
productively with data

69. Scale

70. Secure

71. Software for distributed
storage & analysis
2
Infrastructure for distributed
storage & analysis

72. Amazon EC2

73. Scale out systems
Embarrassingly parallel
Queue based distribution
Small, medium and high scale

74. High performance

75. Compute performance
High performance

76. Cluster Compute
Intel Xeon E5-2670
10 gigabit, non-blocking network
60.5 Gb
Placement groupings

77. Cluster Compute
Intel Xeon E5-2670
10 gigabit, non-blocking network
60.5 Gb
Placement groupings
+GPU

78. 240 TFLOPS

79. Compute performance
High performance
IO performance

80. Unstructured

81. Variable

82. Amazon DynamoDB
Predictable, consistent performance
Unlimited storage
Single digit millisecond latencies
No schema. Zero admin.

83. ...and SSDs for all

84. hi1.4xlarge
2 x 1Tb SSD storage
10 gigabit networking
HVM: 90k IOPS read, 9k to 75k write
PV: 120k IOPS read, 10k to 85k write

85. “The hi1.4xlarge configuration is about half the
system cost for the same throughput.”
Netflix
http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

86. Provisioned IOPS
Provision required IO performance
EBS optimized instances

87. Cost optimization

88. Reserved capacity

89. On-demand
Reserved capacity

90. On-demand
Reserved capacity

91. Spot instances

93. $0.2530 vs $2.40

94. Software for distributed
storage & analysis
2
Infrastructure for distributed
storage & analysis

95. map/reduce

96. Map. Reduce.

97. Write functions. Scale up.

98. Hadoop

99. Undifferentiated heavy lifting

100. Amazon Elastic MapReduce
Managed Hadoop Clusters
Easy to provision and monitor
Write two functions. Scale up.
Choice of Hadoop flavors

101. Amazon Elastic MapReduce
Integrates with S3
Analytics for DynamoDB
Perfect for Spot pricing

102. S3
Input data

103. S3
Input data
Code Elastic
MapReduce

104. S3
Input data
Code Elastic Name
MapReduce node

105. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

106. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

107. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

108. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

109. S3
Input data
Output
S3 + SimpleDB

110. CDC
Centers for Disease Control and Prevention

111. “BioSense 2.0 protects the health of the American people
by providing timely insight into the health of communities,
regions, and the nation by offering a variety of features to
improve data collection, standardization, storage, analysis,
and collaboration”

112. Health data
Collection & storage
Analytics & computation
Collaboration & sharing

113. Health data
highly constrained
Collection & storage
Analytics & computation
Collaboration & sharing

114. HIPAA, HITECH,
FISMA Moderate

115. GovCloud

116. Beyond a definition of Big Data

117. Chromosome 11 : ACTN3 : rs1815739

118. Chromosome X : rs6625163

119. Chromosome 19 : FUT2 : rs601338

120. Chromosome 2 : rs10427255

121. Chromosome 10 : rs7903146
TYPE II

122. Chromosome 15 : rs2472297
+0.25

123. Thank you
matthew@amazon.com
aws.amazon.com
@mza