1. Virtual Sciencein the Cloud Roy Williams California Institute of Technology

2. humans clouds sensors beginner to expert sharing logins and access click to code to workflow personal storage big data and replication compute and scaling software as component interoperabilty survey and event control or autonomous The New Science

3. Compute Services Registry Getting Data

4. Service Oriented Architecture 3. bind service request request client response response 2. find service contract registry 1. publish Principle: Click or Code

5. VO Data Services Cone Search radius+position list of objects encoded as VOTable Simple Image Access Protocol Simple Spectrum Access Protocol spectra have subtleties  protocol more complicated Astronomical Data Query Language For database queries Core SQL functions plus astronomy-specific extensions Sky region, Xmatch Table Access Protocol Exposes relational databases What tables What table schema Here is a query in ADQL

6. VO Compute Services Asynchronous May not get immediate answer just get a place to check back Security Expensive resources, big requests, sequestered data Strong or Weak or None Scalable Graduated path to powerful computation and big data Cloud store VOSpace Sharable

7. VO Registry publish -- find -- bind Registry Metadata Descriptions of data collections data delivery services organizations, etc. Based on Dublin Core with astronomy-specific extensions Represented as XML schema; extensible Contents stored in Resource Registries exchange metadata records through the Open Archives Initiative Protocol (OAI-PMH)

8. Distributed Registry Astrogrid CfA NCSA CDS ESO STScI/JHU NOAO Caltech HEASARC JapanVO Ongoing harvesting March 07 (CfA, ESO, NOAO soon)

9. Semantics & Search Identifiers ivo://nasa.gsfc.gcn/SWIFT#BAT_GRB_Pos_374875-722 Free tags beard Fred pudding Controlled Vocab (UCD) phot.flux;em.ir Controlled Vocabinterop (SKOS) Ontology Greek isA Man, Socrates isA Greek  Socrates isA Man Data Models Each sky position will have a circular positional error estimate ... Text markup Outflows from <object>NGC 666</object> are irregular ... Schema Columns are Magnitude, Position, Identifier , ... Metadata (registry) forms Full Registry: true; ManagedAuthorities: authority, nasa.heasarc Formal service description

10. Cloud Based Tools code & presentation data

14. Query execution

16. Skyalert Push-based workflow Can be cyclic Portfolio aggregation by citation Annotation as software components Stream owner builds template Django, Python, Jquery now 4 developers via SVN

17. Skyalert Stream Registry... will be VO registry

18. Roles human or robot1. browse query, human computing, WWT/Google skyalert.org human or robot2. subscribe human or robot3. author 4. annotate contrib software components archive, mining push inject web portfolios db IM/tweet/email/TCP triggers actions

19. skyalert.org Cyclic workflow graph Trigger CRTS[“Geometry”][“Moon angle”] > 30 and SDSS[“Photoprimary”][“g-magnitude”] < 18 Action annotator followup request dynamically loads module run(triggerEvent, portfolio): <business logic> can build event and inject recursively send message Alerts and event cascade 18

21. Data service from CRTS and Skyalert

22. gets JSON event list via http

23. LSST building skyalert clone

24. Pasadena and Tucson both get events by Jabber/XMPP

25. “Unknown” is now choice ofCataclysmic Variable, Supernova, Blazar Outburst, Active Galactic Nucleus Variability, UVCeti Variable, Asteroid, Variable, Mira Variable, High Proper Motion Star, Comet, Eclipsing Variable, Gamma Ray Burst Afterglow, Microlensing, Nova, Planetary Microlensing, RRLyrae Variable, Tidal Disruption Flare

27. Event ServersTier2 Authoring Distribution Jabber/XMPP or raw socket Tier1: Immediate Forwarding, Reliable?, Topology? Tier2: Subscription, Repository, Query, Portfolio, Registry, Machine Learning, Substreams etc etc

29. 11 Resource Provider sites, >2 Petaflop HPC & >27000 CPUs, >3 Petabyte disk, >60 PB tape

30. Fast network, Visualization, experiments (VMs, GPUs, FPGAs)

32. Architectures 2010 Science Gateway (no architecture!) Node farm (condor) Parallel computing Message-passing MPI Shared memory Graphics Processing Units 104 independent tiny threads Data Intensive Flash memory (TG/UCSD) Graywulf (JHU/Pannstarrs) Immediate resources

33. Science Gateways Biology and Biomedicine Science Gateway Open Life Sciences Gateway The Telescience Project Grid Analysis Environment (GAE) Neutron Science Instrument Gateway TeraGrid Visualization Gateway, ANL BIRN Open Science Grid (OSG) Special PRiority and Urgent Computing Environment (SPRUCE) National Virtual Observatory (NVO) Arroyo Adaptive Optics Linked Environments for Atmospheric Discovery (LEAD) Computational Chemistry Grid (GridChem) Computational Science and Engineering Online (CSE-Online) GEON(GEOsciences Network) Network for Earthquake Engineering Simulation (NEES) SCEC Earthworks Project Network for Computational Nanotechnology and nanoHUB GIScience Gateway (GISolve) Gridblast Bioinformatics Gateway Earth Systems Grid Astrophysical Data Repository (Cornell) Slide courtesy of Nancy Wilkins-Diehr

34. GPU for molecular modelling

35. Pannstarrs PS1 compute User facing SQL/casjobs workbench privacy/share stored queries Data valet load/validate merge crawl replicate log workflow workflow data head/slice hot/warm/cold Fault tolerance: multiple replication, fault workflow Cost and energy carefully considered Future: Hadoop/Mapreduce

36. Cloud Supercomputing? Teragrid/Globusvs Cloud/Amazon MI Both ways to get wholesale computing Both provide IaaS, Infrastructure as a Service Virtual Machine more popular than CTSS stack What about parallelism? I/O speed? GPUs? etc Watch 3leaf and ScaleMP for these

37. Science and Web 2.0 Easy for groups to form and collaborate Integrates with user workspace iGoogle and OpenSocial alongside other aspects of their lives Use existing tools SlideShare, blogs, google gadgets, facebook, Gwave, Flickr, YouTube Sharing workspace Electronic log Provenance Virtual Data as “equivalent script”

38. Science and Web 2.0 Server delivers only code Browser makes presentation Ajax and Ajaj and Http “long poll” Jquery and Google toolkit see WWT and GSky in Skyalert “Everything is a wiki” or a wave? Visible/editable by group/s

40. 30-meter telescope

41. Planet finding coronograph

42. 4-day run for 4-sec!

43. Parallel  parameter sweepsproposed upgrade of the Palomar AO system to a 56x56 subaperture system

44. Arroyo

45. Arroyo Gateway Architecture 1. use HTML/JS from webserver to create job definition. wholesale computing 2. Daemon is polling & sees new job, makes local space for it. 3. Start job on compute resource & update jpb status. daemon 7. User fetches results from webserver 4. Fetch &update status of running job. Repeat. 5. Output to remote space. webserver Django MySQL job definitions and status 5. Daemon copies output from remote to local, updates job status. local space for results remote space for results retail wholesale RW and J. Bunn

46. Pegasus workflow E. Deelman

47. E. Deelman, G. Berriman, RW, et al

49. now 45,000 jobs per month

51. Detailed progress reports during run

53. Wide-area Mosaicking 158 feet Griffith Observatory, Los Angeles

54. Citizen Science

55. Human Volunteers Science Layer Describe what you see in image Each person has level of expertise How to use results most effectively Galaxyzoo.org, citizensky.org good models Game Layer Makes people come back Top 10 ranking etc Anonymous partner a la gwap.com

56. Human Volunteer Evidence Donalek et al arXiv:0810.4945 [astro-ph] 4 of 10 say artifact artifact

57. RW and C. Donalek

58. Macromolecule Citizen Science A. Cunha

59. Information Fusion

60. Classic Machine LearningMetric in “Feature Space” Relevance Vector Machine (Tipping) Feature Vectors Learning from Training set Picking relevant lessons RW and J. Beck

62. Logic handles complexity

63. Probability handles uncertainty

64. Markov Logic Networks

65. Matrix Completion

67. Lessons Learned

68. User Interface (wrong) and now do some science.... Finally get some help Ask for help Translate VOTable format Learn to use VO Registry Read about web services Read about XML Wait for account Register

69. User interface (right) in Darwinian evolution every small change must give benefit Power user Learn the VO structure hey this is interesting .... Run bigger job more science.... Register some science.... Web form Anonymous be careful with complex authentication!

70. Steering the Ship Short term Pragmatism useful tools now simple protocols (eg cone search) “just use RA and Dec” vs Long term Architecture modular suite of interoperable tools sophisticated protocols (egskynode) sophisticated Space-Time coordinates

72. Agreements

73. Data Models

74. Tight Schema

75. Loose Schema

76. UML

77. XSD

78. WSDL

79. Semantics

80. Meaning

81. Usefulness

82. Applicability

83. Code

84. Services

85. InterfacesA Data Model is a bridge from community to computers

86. What is a Data Center? machines services doesn’t matter where or how testing testing testing do we have enough power and HVAC?

87. Complex scienceComplex machines Separate science user from complexity Must have domain science context Making simple things simple but Power to scale up Drill-down if wanted Machines are not the objective Science through data, compute, sharing

88. eScience is for People, right? Getting Started Help Desk Forum Documentation Knowledge Base Calendar Contact Us Social Media Blog/newsfeed Campus Champions Summer Schools Advanced Support for Developers Education