My introductory talk at GlobusWorld 2012, in which I review the progress made with Globus Online during the last year, and introduce our new Globus Storage, Globus Collaborate, and Globus Integrate services. A great demo showed it all working.
3. Run a research project from a coffee shop?
www.globustoolkit.org 3 www.globusonline.org
4. Towards “research IT as a service”
• Dark Energy Survey • SBGrid structural biology consortium
• Galaxy genomics • NCAR climate data applications
• LIGO observatory • Land use change; economics
www.globustoolkit.org www.globusonline.org
5. Let’s rethink how we provide research IT
Accelerate discovery and innovation worldwide by
providing research IT as a service
Leverage the cloud to
• provide millions of researchers with unprecedented
access to powerful tools;
• enable a massive shortening of cycle times in
time-consuming research processes; and
• reduce research IT costs dramatically via
economies of scale
www.globustoolkit.org www.globusonline.org
6. 2011: Grid meets Cloud
Globus Toolkit Globus Online
Build the Grid Use the Grid
Components for building Reliable file transfer
custom grid solutions Software-as-a-Service
globustoolkit.org globusonline.org
www.globustoolkit.org 6 www.globusonline.org
7. Globus Online – The first 18 months
• >4,500 registered users; adding ~300/month
• >3.5 PB moved; averaging ~50 TB/week
Data Transferred (TB/week) Support Requests Received
200 250
200
150
150
100
100
50 50
0 0
Jan 2011 Feb 2012 Dec 2010 Jan 2011
www.globustoolkit.org 7 www.globusonline.org
8. Globus Online – The first 18 months
• 514 active endpoints (year-to-date) including
62 at Tier 1 research institutions
• >390,000,000 files moved
• >100,000 transfer requests
• 99.9% uptime in 2012
• 18 identity providers supported, using multiple
protocols: X.509, MyProxy OAuth, OpenID
www.globustoolkit.org 8 www.globusonline.org
11. 2012: Move. Store. Collaborate.
globus online
Globus Globus Globus
Transfer Storage Collaborate
Globus Integrate Globus Connect Multi User
Globus Connect
Globus Toolkit Globus Nexus
www.globustoolkit.org 11 www.globusonline.org
12. Towards “research IT as a service”
Research Data Management-as-a-Service
Globus Globus Globus Globus …SaaS
Transfer Storage Collaborate Catalog
Globus Integrate …PaaS
www.globustoolkit.org www.globusonline.org
13. Globus Storage: For when you want to …
• Place your data
where you want
• Access it from
Globus Transfer, HTTP/REST, Desktop sync
anywhere via different
protocols
• Update it, version it, Globus
Storage
and take snapshots volume
• Share versions with
who you want
• Synchronize among Commercial National Campus
storage service research computin
locations provider center g center
www.globustoolkit.org www.globusonline.org
14. Globus Collaborate: For when you want to
Join with a few or many people to:
• Share documents
• Track tasks
• Send email
• Share data
• Do whatever
With:
• Common groups
• Delegated
management
www.globustoolkit.org www.globusonline.org
15. Globus Integrate: For when you want to…
Write programs that access/manage user
identities, profiles, groups, resources—and data …
Globus Globus Globus
Transfer Storage Collaborate
Globus Integrate Globus Connect Multi User
Globus Connect
Globus Toolkit Globus Nexus
… via REST APIs and command line programs
www.globustoolkit.org www.globusonline.org
16. Earth System Grid – Portal integration
• Outsource data transfer to Globus
– Data download to user machine
from search
– Data transfer to another server
by user
– Replication of data between sites
by administrator
• No ESGF client software needed
www.globustoolkit.org 16 www.globusonline.org
18. Let’s take a look at what’s coming…
Traumatic Brain Injury (TBI)
Diffusion Tensor Imaging (DTI)
www.globustoolkit.org 18 www.globusonline.org
19. Let’s take a look at what’s coming…
Globus Connect
Bryce Move DTI results to PADS
Bryce’s laptop Compute
DTI Group Cluster
- Kyle
- Bryce Globus Storage Globus Transfer
Create snapshot to Copy TBI data to
share with group compute cluster
Globus Nexus Globus Transfer
Add Bryce to TBI Move DTI results
collaboration to shared volume
Globus Collaborate
Publish DTI data to TBI
web site
Amazon S3
Globus Storage
Create volume and
share with TBI group SDSC
UChicago
Cloud
Kyle “TBI” Object
Globus Connect volume Store
Move MRI files to Cornell
TBI shared volume Red Cloud
www.globustoolkit.org 19 www.globusonline.org
20. Globus Toolkit update
GT Releases 2011-2012
5/4/11 GT 5.1.0
5/18/11 GT 5.0.4
8/2/11 GT 5.1.1 Highlights
11/16/11 GT 5.1.3 (5.2 beta) • Native packaging
12/15/11 GT 5.2.0 • GridFTP: DCSC support
3/8/12 GT 5.0.5 • New GRAM5 version;
Likely last 5.0.x release in use on OSG
4/5/12 GT 5.2.1rc1
5.2.1 final planned for April
www.globustoolkit.org 20 www.globusonline.org
21. Globus Toolkit update – GridFTP
• GT 5.2.0
– Support for DCSC command
• Allows client to specify credentials used to secure data
channel connection
• Utilized by Globus Transfer for seamless data movement
across multiple security domains
– Server administrators may now restrict client access
to a set of paths
• Coming in 5.2.1
– MLSC – stream directory listings over control channel
• Enables simple (control channel-only) implementation of
Globus Storage ftp server
• Can help directory listings for gridftp servers behind
www.globustoolkit.org restrictive firewalls 21 www.globusonline.org
22. Delivery plans
Globus Transfer Globus Storage Globus Collaborate
• Generally available • Early release • Initial projects at
• Service and Web UI • Generally available in UChicago
enhancements Fall 2012 • Early release sometime
continue before year-end 2012
Globus Integrate
Globus Connect Multi User (GA)
• Transfer API available
• APIs for Storage, Collaborate
planned after app release Globus Connect (GA)
Globus Toolkit (GA V5.2) Globus Nexus (Alpha)
www.globustoolkit.org www.globusonline.org
23. Delivery plans
• Globus Storage
– Current: Early access – contact us if interested
– Open beta planned in late Spring; GA in late Summer
– V1.0: UChicago and Amazon S3 object stores
• Globus Collaborate
– Current: Pre-release with two UChicago/CI groups
– First release planned around year-end
www.globustoolkit.org 23 www.globusonline.org
24. Premium offerings
Motivated by sustainability: resource cost
recovery, user support, future development
• Globus Transfer plans (resource providers)
– www.globusonline.org/premium-plans/
• Globus Storage premium plans
– Plans variables include: size, object store
type, durability, support level, personalization
• Globus Collaborate premium plans
– Plans variables include: branding capabilities, support
level, integration (e.g. identity)
www.globustoolkit.org 24 www.globusonline.org
25. Premium offerings
Motivated by sustainability: resource cost
recovery, user support, future development
• Globus Transfer plans (resource providers)
– www.globusonline.org/premium-plans/
• Globus Storage premium plans
– Plans variables include: size, object store
type, durability, support level, personalization
• Globus Collaborate premium plans
– Plans variables include: branding capabilities, support
level, integration (e.g. identity)
www.globustoolkit.org 25 www.globusonline.org
26. The Globus yeam
Bryce Allen Rachana Lisa
Ananthakris Childers
hnan
Martin Tom Howe Raj Jack Kordas Lukasz
Feller Kettimuthu Lacinski
Lee Liming JP Navarro Karl Pickett
Andrew Peter Zich
Zich
www.globustoolkit.org 26 www.globusonline.org
Genome (x10,000 in 6 years!), astronomy, social sciences, HPC …Big projects, e.g., LHC, have things under controlWhat about small and medium labs? They are struggling
Small and medium businesses deal with these challenges by outsourcing many business functions to third-party “cloud” providers.Last year we introduced the seemingly farfetched notion that one could run a research project from a coffee shop…
We have been investigating over the past few years just what those services may be.
Not (particularly) computing as a serviceBut the IT functions that researchers need to functionInclude collaboration as a service
An example of integrating Globus services, in this case Globus Transfer, using Globus NexusMention branding is/will be available for every Globus service100 GB/sec
Moving beyond Transfer service… introduce planned services hereAlso introduce the new naming hereToday: Globus Online Globus Transfer; moving to Globus (+ Service Name)