1. Platforms for
Data Science
Deepak Singh
P r i n c i p a l P r o d u c t M a n a g e r

4. bioinformatics

5. collection

6. curation

7. analysis

8. so what?

9. Image:Yael Fitzpatrick (AAAS)

10. lots of data

11. lots of people

12. lots of places

13. constant change

14. we want to make our
data more effective

15. versioning

16. provenance

17. filter
Via asklar under a CC-BY license

18. aggregate
Image: Chris Heiler

19. extend
Image: Bethan

20. human interfaces
Image: Sebastian Anthony

21. share

22. communicate
image: Leo Reynolds

23. hard problem

24. really hard problem

25. so how do
get there?

26. information
platforms

27. Image: Drew Conway

28. dataspaces
Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data

29. the unreasonable
effectiveness of data
Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)

30. accept all data
formats

31. evolve APIs

32. data as a
programmable
resource

33. data is a
royal garden

34. compute is a
fungible commodity

35. constraints
everywhere

36. CPU, storage,
Hardware
memory
Collections, datasets,
Data management
provenance
parallelization,
Software
optimization
Backup, redundant,
Availability
replicated
Cost Small

37. remove constraints
Credit: Pieter Musterd a CC-BY-NC-ND license

38. amazon web services

39. your
infrastructure

41. ec2-run-instances

43. on demand
global
secure

44. programmable

48. elastic

49. Netflix needed to transcode
17,000 titles (80TB of data) to
support the launch of Sony PS3.
They provisioned 1200 Amazon
EC2 instances and completed
the transcoding process in just
days.
Source: Adrian Cockroft (Netflix)

50. Source: Adrian Cockroft (Netflix)

51. durable

52. 99.999999999%

53. I did say data was a
royal garden

54. performance

55. “Our 40-instance (m2.2xlarge) cluster can
scan, filter, and aggregate 1 billion rows in
950 milliseconds.”
Mike Driscoll - Metamarkets

56. WIEN2K Parallel
Performance
H size 56,000 (25GB)
Runtime (16x8 processors)
Local (Infiniband) 3h:48
Cloud (10Gbps) 1h:30 ($40)
1200 atom unit cell; SCALAPACK+MPI
diagonalization, matrix size 50k-100k
Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)

57. “Our tests have shown more than 90
percent scaling efficiency on
clusters of up to 128 GPUs each”

58. consumption
models

59. on-demand

60. Reserved Instances

61. what is the value of
your data?

64. the clouds
biggest value

65. remove constraints

67. Image: Chris Dagdigian

68. Credit: Angel Pizzaro, U. Penn

69. 13k sequences
- 10 min
- 0.1s per sequence

70. mapreduce for
genomics
http://bowtie-bio.sourceforge.net/crossbow/index.shtml
http://contrail-bio.sourceforge.net
http://bowtie-bio.sourceforge.net/myrna/index.shtml

74. 30,472 cores

75. $1279/hr

76. http://cloudbiolinux.org/

77. http://usegalaxy.org/cloud

78. “The process of moving StarMolsim over to the cloud to support the “Introduction to
Modeling and Simulation” course at MIT was a huge success. The cloud enabled the STAR
group to move away from the responsibility of owning and maintaing dedicated hardware
and instead focus on their core mission of developing software and services for faculty,
students, and researchers at MIT”
http://web.mit.edu/stardev/cluster/about.html

79. in summary

80. large scale data
requires a rethink

81. data architecture

82. compute architecture

83. distributed,
programmable
infrastructure

84. amazon web services

85. remove constraints

86. can we build data
science platforms?

87. there is no magic
there is only awesome

88. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.com
Inspiration and ideas from
Matt Wood& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license