Science Gateways and their framework presented ad eResearch 2013 conference

1. Capetown, South Africa– 04/10/2013
Riccardo Rotondo – Consortium GARR
(riccardo.rotondo@garr.it)
Science Gateways: one portal,
many e-Infrastructures and
related services

2. Outline
• Did HPC make the world flat?
• Science Gateways
– Overview
– CTSG Framework
• Services
• Use case
• Training activities
– Material available
– Join us: Marketplace
Riccardo Rotondo 2October 8th, 2013

3. Is the world flat?
Riccardo Rotondo 3October 8th, 2013

4. The world is flat
Riccardo Rotondo 4October 8th, 2013
In 2006, Thomas
Friedman's book The
World is Flat portrayed a
global population that
was more borderless and
interconnected than ever
before.
Bibliographic information
ISBN 1429923075, 9781429923071

5. Grid & Cloud Computing
Riccardo Rotondo 5October 8th, 2013

6. Riccardo Rotondo 6
Users in the net
October 8th, 2013

7. Some barriers limit grid
adoption(*)
Riccardo RotondoOctober 8th, 2013
The eResearch2020 report
http://www.eresearch2020.eu/
7

8. Did we learn from Grid ?
Riccardo Rotondo 8October 8th, 2013

9. Reusability
Riccardo Rotondo 9October 8th, 2013

10. Easy of use and access
Riccardo Rotondo 10October 8th, 2013

11. Science Gateways
Riccardo Rotondo 11October 8th, 2013
“A Science Gateway is a community-developed set of
tools, applications, and data that is integrated via a
portal or a suite of applications, usually in a graphical
user interface, that is further customized to meet the
needs of a specific community.”
Teragrid/XSEDE
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013

12. Science Gateway’s Requirements
 Authentication and Autorisation
 SAML, LDAP
 Application e-Infrastructure
indipendent
 jSAGA, SAGA, OCCI
 Standard Java Technology
 JSR 168/286
 Web Technology
 Web CMS
 Wiki, Blog, Messages Board, Vconf, Adobe Connect
 Portal Framework
 Liferay portlet container
 Liferay Service Builder
Riccardo Rotondo 12October 8th, 2013

13. Catania Science Gateway Framework architecture
Riccardo Rotondo 13October 8th, 2013
.......
CataniaScience
Gateway
App. 1 App. 2 MyCloud
Embedded Services Administrator(s)
Scientists
Cloud tenants
Users belonging
to Identity
Federations
Grid/Cloud
Engine
(based on SAGA)
CLEVER
Orchestrator
(based on OCCI)
Cloud #2 Cloud #n
Cloud #1
Single logical domain
HPC
Clusters
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013

14. Catania Science Gateway Framerwork
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 14October 8th, 2013

15. AuthN/AuthZ Schema
15
e-Infrastructures
Federation
2. Forwarded
to the IdP
Retrieve e-Infrastructure
credentialsScience Gateway
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo

16. Federated User
Riccardo Rotondo 16October 8th, 2013
Science
Gateway

17. Social User
Riccardo Rotondo 17October 8th, 2013
Science
Gateway

18. Enhancements of user management
Riccardo RotondoOctober 8th, 2013
• Liferay supports several system to store users
data, both local and remote
• Supporting the largest number of users in the
easiest way
• A modular way to distinguish between
different services and privileges is need
• Science Gateways stores users on an LDAP
server
• In the act of registration user data must be
written on the LDAP connected to Liferay
• A portlet has been developed to perform this
actions
18

19. Authentication request on Africa GRID SGW
Riccardo Rotondo 19October 8th, 2013

20. Mobile Authentication
20
Native apps
Web views
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo

21. Some of the IdPs supported
21
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo

22. Users currently registered
(as of the end of 2012)
22
Users from 219 Organisations in 47 Countries
12 Science Gateways in production; others in preparation
October 8th, 2013 Riccardo Rotondo

23. Applications accessing grid services
Riccardo Rotondo 23October 8th, 2013
• 12 applications developed among 5
different countries and 3 continents
(Europe, Latin America and Asia);
• 4 scientific domains:
– Life Science;
– Mathematic & Computer Science;
– High Energy Physics;
– Cultural Heritage.

24. Integrated Services
Riccardo Rotondo 24October 8th, 2013
GRIDCLOUD
JSR
168/286
JSR
168/286
JSR
168/286
JSR
168/286
Catania Science Gateway Framework
Local Cluster

25. CSGF components
Riccardo Rotondo 25October 8th, 2013
Grid Engine e-Infrastructure
& Data Service
Cloud Services
JSAGA
Adaptors

26. Catania Science Gateway Framerwork
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 26October 8th, 2013

27. A Simple API for Grid Applications (SAGA)
Riccardo Rotondo 27October 8th, 2013
 SAGA is an API that provides the basic
functionality required to build distributed
applications, tools and frameworks;
 It is independent of the details of the
underlying infrastructure (e.g., the
middleware);
 SAGA is an OGF specification:
http://www.gridforum.org/documents/GFD.90
.pdf.

28. A Simple API for Grid Applications (SAGA)
Riccardo RotondoOctober 8th, 2013
 SAGA is composed by:
 SAGA Core Libraries: containing the SAGA base
system, the runtime and the API packages (file
management, job management, etc.);
 SAGA Adaptors: libraries providing access to the
underlying grid infrastructure (adaptors are
available for Globus, gLite, etc.);
 SAGA defines a standard
We then need an
implementation!
28

29. A Simple API for Grid Applications (SAGA)
Riccardo RotondoOctober 8th, 2013
 Several Implementations are available:
 A C++ and a Java implementation developed at
the Louisiana State University / CCT and Vrije
Universiteit Amsterdam (http://saga.cct.lsu.edu);
 A Java implementation developed at CCIN2P3
(http://grid.in2p3.fr/jsaga/);
 A Python implementation based on those above.
29

30. Job Engine
Middleware Independent
Riccardo RotondoOctober 8th, 2013
JSAGA supports gLite, Globus,
ARC, UNICORE, etc.
30

31. Job Engine - Architecture
Riccardo RotondoOctober 8th, 2013
WT
Worker Threads for Job Submission
WT
Worker Threads for
Job Check Status
USERS
TRACKING
DB
MONITORING
MODULE
Resources
Jobs Queue
WT WT
WT WT WT
WT
WT WT
Jobs
Submission
Jobs
Check status/
Get output
31

32. Job Engine at work
Riccardo RotondoOctober 8th, 2013
1. Sign in
eTokenServer
User
Track-
ing DB
5. e-Infra
Interactions
5.
Tracking
2. Grid
Request
6. Getting
Results
* or equivalent e-Infra auth
32

33. Job Submission
Riccardo RotondoOctober 8th, 2013 33

34. Job Submission
Riccardo RotondoOctober 8th, 2013 34

35. My Workspace – Active Job List
Riccardo RotondoOctober 8th, 2013 35

36. Accessing services from Facebook
Riccardo Rotondo 36October 8th, 2013

37. Catania Science Gateway Framerwork
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 37October 8th, 2013

38. Motivations
Um… isn’t your
computer on fire
?
It’s ok, my files
are stored in a
safer place.
Image source:
October 8th, 2013 Riccardo Rotondo 38

39. e-Infrastructure & Data Service
• We designed the architecture to provide a
Service accessible from your application.
• Modular attachable resources available to
the SG thanks to the straight coupled
service that allows read/write data from
several e-Infrastructures.
Riccardo Rotondo 39October 8th, 2013

40. e-Infrastructure Service
• Four different kind of e-Infrastructure
identified:
– Local
– Remote
– Grid
– Cloud
• Possibility of defining different Policy and
Quota according e-infrastructures/users
availability/needs
• Default and favorite e-Infrastructures
selectable
• Disaster recovery and (cross) file replica.
Riccardo Rotondo 40October 8th, 2013

41. e-Infrastructure Service
• If you want to copy a file on a file
system you plug-in an external storage
to it.
Riccardo Rotondo 41October 8th, 2013
HFS
ext4
NFS
FAT

42. e-Infrastructure Service
• On a SG now you are able to plug-in an
e-Infrastructure.
Riccardo Rotondo 42
Local Storage
Remote Storage
Grid Storage
Cloud Storage
October 8th, 2013

43. Data Services
• Business logic related to file management
totally independent from the underneath
technology
• All the common features associated to file
provided: create, copy, delete, move and
sharing.
• For each features, we implemented a different
Storage Interface that translates the single
commands according to the technology
• All data are stored in different tables in order
to let users organise files on a virtual file
system.
Riccardo Rotondo 43October 8th, 2013

44. Data Service
Riccardo Rotondo 44
Local Storage
Remote Storage
Grid Storage
Cloud Storage
Different
implementations
of the same
storage interface
October 8th, 2013

45. Upload
workflow
Riccardo Rotondo 45
1.
Sign
in
Database
5. File
Upload
4. e-Infra
found
7. Update
6. Upload on
e-Infras
2. Upload
request
3. e-Infra
Discovery

46. Data Management on various
e-Infrastructures
46October 8th, 2013 Riccardo Rotondo

47. Catania Science Gateway Framework
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 47October 8th, 2013

48. My Cloud (cloud interoperability based on OCCI Standard)
Riccardo Rotondo 48October 8th, 2013
For more information
follow Roberto Barbera
presentation

49. Science Gateways deployed
49
12 SGs in production and
others in development
Very easy and intuitive access procedure
User-driven development
Surveys to propose applications are available in
Italian and other languages
VRCs supported either by
region or discipline
Africa Grid
agINFRA
CHAIN-REDS
COGITO-MED
DCH-RP e-Culture
DECIDE
EarthServer
EUMEDGRID
GARR
GISELA
IGI
KLIOS
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo

50. October 8th, 2013 Riccardo Rotondo
Training activities
50

51. SG under construction
Riccardo Rotondo 51October 8th, 2013

52. Web-course summary information
• Home page:
– http://agenda.ct.infn.it/event/catania-sg-
course
• Timetable (direct URL):
– http://agenda.ct.infn.it/conferenceOtherViews.
py?view=standard&confId=942
• All presentations have been uploaded
• Links are provided both to watch and
download recordings of all the webinars
October 8th, 2013 Riccardo Rotondo 52

53. Training material
• Training material page (short URL):
– http://goo.gl/im7YG
• Available sections
– SG plugin as portlets accessing e-Infrastructure
services;
– Multi federation discovery service
– Identity Provider installation an configuration
October 8th, 2013 Riccardo Rotondo 53

54. Our code on sourceforge
• The Catania Science Gateway Framework
on Sourceforge:
– http://sourceforge.net/projects/ctsciencegtwys/
• Wiki
• Tickets
• Discussion
• Code is open source, feel free to join and
contribute
• For more info contact to sg-licence@ct.infn.it
October 8th, 2013 Riccardo Rotondo 54

55. A unique entry point
• The Catania Science Gateway Framework on the web:
– http://www.catania-science-gateways.it
October 8th, 2013 Riccardo Rotondo 55

56. 1.
2.
3.
4.
5.
Users/VRCs
Ex.: Africa Grid Science Gateway
SG Dev.
The Science Gateway Market Place
Riccardo RotondoOctober 8th, 2013 56

57. The list of available applications
October 8th, 2013 Riccardo Rotondo 57

58. To propose new applications
• http://www.catania-science-gateways.it/survey
October 8th, 2013 Riccardo Rotondo 58

59. To port new applications
• Register as a Catania Science Gateway Developer:
– http://www.catania-science-gateways.it/port-app
• Sign In and book an application «to be ported»
• Once your booking is accepted, you have 30 days to create the needed
portlet(s) for the application (the sooner the better)
– After that time the access to our VPN will be denied
October 8th, 2013 Riccardo Rotondo 59

60. Conclusion
• Science Gateways permit easy federated
access to e-Infrastructure services in a
standards-oriented way
• We are developing a modular framework to
increase the number of applications
supported within our portals
• All functionalities are designed according
users needs who give us inspiration and
contributors … the next could be you!
Riccardo Rotondo 60October 8th, 2013

61. Questions ?
Riccardo Rotondo 61October 8th, 2013