This is a talk titled "Cloud-Based Services For Large Scale Analysis of Sequence & Expression Data: Lessons from Cistrack" that I gave at CAMDA 2009 on October 6, 2009.

1. Cloud-Based Services For Large Scale Analysis of Sequence & Expression Data: Lessons from Cistrack October 6, 2009 Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago Open Data Group Institute for Genomics & Systems BiologyUniversity of Chicago 1

3. Jia Chen

5. Michal Sabala

6. Damian Roqueiro

7. Parantu K Shah

8. FengTian

10. Projected sequencing capabilities (world-wide) 2060 Total human population 2019 One of each described species 2031 One of each species ~100M estimate log10 billions of base pairs 2023 One of each species ~10M estimate Kevin White, unpublished

11. Is Biology a Large Data Science? vs CPUs double approximately every 18 months (Moore’s Law). Disks double every 12-15 months (Johnson’s Law). Amount of publically available sequence data is doubling approximately every 12 months. 5

12. IBM joins race for $100 personal genome.

13. We Have a Problem vs More and more of your colleagues (e.g. the biologist down the hall) with access to modern instruments are producing so much data that they cannot easily manage, analyze and archive it. Large projects build their own infrastructure. Almost all other biologists are on their own. 7

14. Point of View To do research today… Analytic infrastructure Analytic algorithms & statistical models Data

15. Part 2What is a Cloud? 9

16. What is a Cloud? 10 Software as a Service

17. Is Anything Else a Cloud? 11 Infrastructure as a Service – based upon scaling Virtual Machines (VMs)

18. Are There Other Types of Clouds? 12 ad targeting Large Data Cloud Services

19. What is Virtualization? 13

20. Idea Dates Back to the 1960s 14 App App App CMS CMS MVS IBM VM/370 IBM Mainframe Native (Full) Virtualization Examples: Vmware ESX Virtualization first widely deployed with IBM VM/370.

21. One Definition Clouds provide on-demand resources or services over a network, often the Internet, with the scale and reliability of a data center. No standard definition. Cloud architectures are not new. What is new: Scale Ease of use Pricing model. 15

22. 16 Scale is new.

25. 2004 10x-100x 1976 10x-100x data science 1670 250x simulation science 1609 30x experimental science

26. 20 Clouds vs Grids

27. Hadoop & Sector 21

28. MalStone B Benchmark 22

29. Part 3Cistrack 23 www.cistrack.org

30. Cistrack Resource for cis-regulatory data. It is open source and based upon CUBioS. Currently used by the White Lab at University of Chicago for managing ModENCODE fly data. Contains raw data, intermediate, and analyzed data from approximately 240 experiments from Agilent, Affy and Solexa platforms.

31. Chromatin Developmental Time-Course H3K4me1 enhancers H3K4me3 promoters & enhancers H3K9Ac activation H3K9me3 heterochromotin H3K27Ac activation H3K27me3 repression PolII transcript. & promoters CBP HAT- enhancers Total RNA expression X 12 time points for which chromatin and RNA have been collected (Carolyn Morrison & Nicolas Negre) 8 antibodies used for ChIP experiments (Zirong Li and Bing Ren)

32. 1. Cistrack Supports Cubes of Data Drosophila regulatory elements from Drosophila modENCODE. ChIP-chip data using Agilent 244K dual-color arrays. Six histone modifications (H3K9me3, H3K27me3, H3K4me3, H3K4me1, H3K27Ac, H3K9Ac), PolII and CBP. Each factor has been studied for 12 different time-points of Drosophila development.

33. 2. ChIP-Seq Data Volumes are Large Cistrack integrates with large data clouds.

34. 3. Continuous Reanalysis is Desirable In general, it is quite labor intensive to reanalyze your existing data with a new algorithm. Cistrack supports VMs that can simplify re-applingCistrackpipelines that have been updated to include a new algorithm.

35. Cistrack Architecture Cistrack Web Portal & Widgets Cistrack Database Analysis Pipelines & Re-analysis Services CistrackCloud Services Ingestion Services

36. Part 4Reanalysis 30 Can you repeat an analytic pipeline one year after a post-doc leaves your lab?

37. Promoters: Use H3K4me3, PolII &RNA to Map Active Genes

38. Promoters: Use of H3K4me3, PolII & RNA to Map Active Genes

39. Active Genes - Solexa Result

40. Basic Idea Replace Cloud VM VM VM At anytime, you can launch one or virtual machines (VMs) to redo pipeline analysis and persist results to a database and VM to a cloud.

41. Raywulf We have designed a cluster (called a Raywolf Cloud) that is optimized to serve as your own private cloud. About $2K/TB. Will be used by the Open Science Data Cloud.

42. Acknowledgements

43. Cis-Regulatory Map of the Drosophila Genome (modENCODE) Data Generation Kevin White, U. Chicago (Antibody pipeline, ChIP-chip pipeline) Bing Ren, UCSD (Antibody validation, ChIP-chip pipeline) Robert Grossman U. Illinois (LIMS, data management & analysis) Computational identification of Cis-Regulatory Motifs ManolisKellis, MIT (Motif analysis, ChIP-chip data analysis) Biological validation Jim Posakony, UCSD (Promoters/Enhancers) Steve Russell, Cambridge U. (Insulators/Silencers) Hugo Bellen, Baylor (Element “necessity” validations)

45. Nick Bild

46. Jia Chen

47. Xiangjun Liu

48. Nicolas Negre

49. Damian Roqueiro

50. Parantu K Shah

51. FengTian