Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Grid computing

Grid Computing concepts

  • Inicia sesión para ver los comentarios

  • Sé el primero en recomendar esto

Grid computing

  1. 1. Ramraj Choudhary
  2. 2.  Session – 1 Introduction to Grid Computing  Session – 2 OGSA  Session – 3 OGSI  Session – 4 Globus Toolkit v3
  3. 3. Session - 1
  4. 4.  Introduction  Virtual Organization  Grid Computing Benefits  Industry Participation  Computational Grid Projects  National Fusion Grid  Summary
  5. 5.  “… a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities”  Criteria for a Grid  Coordinated resources that are not subjected to centralized control  Use standard, open, general-purpose protocols  Delivers non-trivial qualities of service
  6. 6. “Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations”
  7. 7. 1800 Physicists, 150 Institutes, 32 Countries 100 PB of data by 2010; 50,000 CPUs?
  8. 8. Tier2 Centre ~1 TIPS Online System Offline Processor Farm ~20 TIPS CERN Computer Centre FermiLab ~4 TIPSFrance Regional Centre Italy Regional Centre Germany Regional Centre InstituteInstituteInstitute Institute ~0.25TIPS Physicist workstations ~100 MBytes/sec ~100 MBytes/sec ~622 Mbits/sec ~1 MBytes/sec There is a “bunch crossing” every 25 nsecs. There are 100 “triggers” per second Each triggered event is ~1 MByte in size Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Physics data cache ~PBytes/sec ~622 Mbits/sec or Air Freight (deprecated) Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Caltech ~1 TIPS ~622 Mbits/sec Tier 0 Tier 1 Tier 2 Tier 4 1 TIPS is approximately 25,000 SpecInt95 equivalents
  9. 9.  Dynamic set of individuals/institutions defined around a set of resource-sharing rules and conditions  Share some commonality  Conditional, time bound, and rules driven resource sharing  Dynamic collection of individuals
  10. 10.  Peer to peer sharing relationship among participant  Assign resources to users from different domains
  11. 11.  Exploit underutilized resources  Resource balancing  Virtualizes resources across an enterprise  Enable collaboration for virtual organizations
  12. 12.  Avaki  Axceleon  CapCal  Centrata  DataSynapse  Distributed Science  Elepar  Entropia.com  Grid Frastructure  GridSystems  Groove Networks  IBM  Intel  Powerllel  ProcessTree  Sharman Networks Kazza  Sun Gridware  Sysnet Solutions  Tsunami Research  Ubero  United Devices  Veritas  Xcomp  Jivalti  Mithral  Mind Electric  Mojo Nation  NewsToYou.com  NICE, Italy  Noemix, Inc.  Oracle  Parabon  Platform Computing  Popular Power Source: http://www.gridcomputing.com/
  13. 13.  Particle Physics  Global sharing of data and computation  Astronomy  ‘Virtual Observatory' for multi-wavelength astrophysics  Chemistry  Remote control of equipment and electronic logbooks  Engineering  Industrial healthcare and virtual organizations  Bioinformatics  Data integration, knowledge discovery and workflow  Healthcare  Sharing normalized mammograms  Environment  Ocean, weather, climate modeling, sensor networks
  14. 14.  A collaborative pilot project that creates and deploys collaborative software tools throughout the magnetic fusion research community
  15. 15.  To advance scientific understanding and innovation in magnetic fusion research by enabling more efficient use of existing experimental facilities and more effective integration of experiment, theory, and modelling.  To advance scientific understanding and innovation in fusion research  Making widespread use of Grid technologies  http://www.fusiongrid.org
  16. 16.  Data, Codes, Analysis Routines, Visualization Tools should be thought of as network accessible services  Shared security infrastructure  Collaborative nature of research requires shared visualization applications and widely deployed collaboration technologies  Integrate geographically diverse groups  Not focused on CPU cycle scavenging or “distributed” supercomputing (typical Grid justifications)  Optimize the most expensive resource - people’s time
  17. 17.  Problem  Data sharing and rapid data analysis among the National Fusion Collaborators Community  Solution  Common data acquisition and management system  Common relational database run-management schema
  18. 18. Geographically Diverse Community  3 Large Experimental Facilities  Alcator, C-Mod, DIII-D  NSTX ~$1B replacement cost  40 U.S. fusion research sites  Over 1000 scientists in 37 state  Efficient collaboration is a requirement!  Integrate geographically diverse groups  One future worldwide machine  Not based in US  US needs collaboration tools to benefit
  19. 19.  Produced both design and architecture documents for review by public (introduction into GGF document process)  Demonstrated full-featured prototypes in Nov 2002 at SC2002  Architecture, venue client, workspace docking complete with application sharing
  20. 20.  Dynamic resource discovery  Resource selection and allocation  Implementing quality of service  Resource monitoring and management  Security  Data management  Information management
  21. 21. Grid Book  www.mkp.com/gri ds The Globus Project™  www.globus.org Global Grid Forum  www.gridforum.o rg TeraGrid  www.teragrid.org EU DataGrid  www.eu- datagrid.org GriPhyN  www.griphyn.org iVDGL  www.ivdgl.org Background papers  www.mcs.anl.gov/~fos ter
  22. 22.  Like the Web – grid computing keeps complexity hidden; multiple users enjoy a single unified experience  Unlike the Web – grid enables not only communication but full collaboration towards common business goals  Like peer-to-peer – grid computing allows users to share files
  23. 23.  Unlike peer-to-peer – grid computing allows many to many sharing not only files but other resources as well  Like clusters and distributed computing – grids bring computing resources together  Unlike cluster and distributed computing – resources can be geographically distributed and heterogeneous
  24. 24.  Like virtualization technologies – grid computing enables the virtualization of IT resources  Unlike virtualization technologies – grid computing enables the virtualization of vast and disparate IT resources
  25. 25.  Introduction  Virtual Organization  Grid Computing Benefits  Industry Participation  Computational Grid Projects  National Fusion Grid  Reference Materials
  26. 26. Session - 2
  27. 27.  Introduction to Globus Project  API and Protocols  Layered Architecture  Web services  Open Grid Service Architecture  Summary
  28. 28.  Home page  http://www.globus. org  Major Contributors  Argonne National Labs  University of Chicago  University of Southern California  Northern Illinois University  Sponsors  DARPA  US Department of Energy  National Science Foundation  NASA  Supporters  IBM, Microsoft, Cisco
  29. 29.  Close collaboration with real Grid projects in science and industry  The Globus Toolkit®: Open source software base for building Grid infrastructure and applications  Development and promotion of standard Grid protocols to enable interoperability and shared infrastructure  Development and promotion of standard Grid software APIs to enable portability and code sharing  Global Grid Forum: Co-founded GGF to foster Grid standardization and community
  30. 30.  A specification for a set of routines to facilitate application development  Refers to definition, not implementation  Often language-specific (or IDL)  Routine name, number, order and type of arguments; mapping to language constructs  Behavior or function of routine  Examples of APIs  GSS-API (security), MPI (message passing)
  31. 31.  A formal description of message formats and a set of rules for message exchange  Rules may define sequence of message exchanges  Protocol may define state-change in endpoint, e.g., file system state change  Good protocols designed to do one thing  Protocols can be layered  Examples of protocols  IP, TCP, TLS (was SSL), HTTP, Kerberos
  32. 32. TCP/IP APIs include BSD sockets, Winsock, System V streams, … The protocol provides interoperability: programs using different APIs can exchange information I don’t need to know remote user’s API TCP/IP Protocol: Reliable byte streams WinSock API Berkeley Sockets API Application Application
  33. 33. An API provides portability: any correct program compiles & runs on a platform Does not provide interoperability: all processes must link against same SDK  E.g., MPICH and LAM versions of MPI ApplicationApplication MPI API MPI API LAM SDK LAM protocol MPICH-P4 SDK MPICH-P4 protocol TCP/IP TCP/IP Different message formats, exchange sequences, etc.
  34. 34. Application Fabric “Controlling things locally”: Access to, & control of, resources Connectivity “Talking to things”: communication (Internet protocols) & security Resource “Sharing single resources”: negotiating access, controlling use Collective “Coordinating multiple resources”: ubiquitous infrastructure services, app- specific distributed services Internet Transport Application Link InternetProtocolArchitecture “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, Foster, Kesselman, Tuecke, Intl Journal of High Performance Computing Applications, 15(3), 2001.
  35. 35.  Fabric layer  Defines logical /physical resources  Physical resources – computational resources  Logical resources – distributed file systems  Can be implemented by their own protocol  Capabilities of resources  Provide an inquiry mechanism for discovery  Provide resource management capability for resource provisioning
  36. 36.  Connectivity layer  Defines core communication and authentication protocols  Authentication solution for VO’s  Single sign on  Delegation – ability to access a resource under current user permissions  Integration with local resource specific security solutions  User based trust relationship
  37. 37.  Resource layer  Controls the secure negotiation, initiation, monitoring, sharing of operations across individual layer  Primary resource layer protocol  Negotiating access to shared resource  Performing operation on a resource and monitoring the status of operation
  38. 38.  Collective layer  Responsible for global resource management  Common collective services in a Grid computing System  Discovery services  Co-allocation, scheduling services  Community accounting and payment services
  39. 39.  Application layer  User applications constructed by utilizing the services defined at each lower level  Each layer in the Grid architecture provides a set of API’s and SDK’s for the higher layers of integration
  40. 40.  Focus on architecture issues  Propose set of core services as basic infrastructure  Use to construct high-level, domain-specific solutions  Design principles  Keep participation cost low  Enable local control  Support for adaptation  “IP hourglass” model Diverse global services Core services Local OS A p p l i c a t i o n s
  41. 41.  The Globus Toolkit v2 (GT2) centers around four key protocols – Connectivity layer: >Security: Grid Security Infrastructure (GSI) – Resource layer: >Resource Management: Grid Resource Allocation Management (GRAM) >Information Services: Grid Resource Information Protocol (GRIP) >Data Transfer: Grid File Transfer Protocol (GridFTP)  Also key collective layer protocols – Info Services, Replica Management, etc.
  42. 42.  Heterogeneous protocol base  Increasing number of virtual services that needed to be managed  Web services (WSDL, SOAP) appeared
  43. 43.  At the heart of Web services is:  WSDL: Language for defining abstract service interfaces  SOAP (and friends): Binding from WSDL to bytes on the wire  Web services appears to offer a fighting chance at ubiquity (unlike CORBA)  But Web services does not go far enough to serve a common base for the Grid…
  44. 44.  Interface to a well-explained, self-contained function (module, component)  Usu. small grained  Defined in WSDL  Communication: XML and SOAP OnlineStore Web service queryItems() orderItem() Web service client SOAP/XML
  45. 45. OnlineStore WS queryItems() orderItem() Web service client StockMgr WS query() ship() OrderMgr WS order() bill() query order ship warehouse
  46. 46.  XML-based distributed computing technology  Web service = a server process that exposes typed ports to the network  Described by the Web Services Description Language, an XML document that contains  Type of message(s) the service understands & types of responses & exceptions it returns  “Methods” bound together as “port types”  Port types bound to protocols as “ports”  A WSDL document completely defines a service and how to access it
  47. 47. <wsdl:definitions targetNamespace=“…”> <wsdl:types> <schema> <xsd:element name=“fooInput” …/> <xsd:element name=“fooOutput” …/> </schema> </wsdl:types> <wsdl:message name=“fooInputMessage”> <part name=“parameters” element=“fooInput”/> </wsdl:message> <wsdl:message name=“fooOutputMessage”> <part name=“parameters” element=“fooOutput”/> </wsdl:message> <wsdl:portType name=“fooInterface”> <wsdl:operation name=“foo”> <input message=“fooInput”/> <output message = “fooOutput”/> </wsdl:operation> </wsdl:portType> </wsdl:definitions>
  48. 48.  “Web services” address discovery & invocation of persistent services  Interface to persistent state of entire enterprise  In Grids, must also support transient service instances, created/destroyed dynamically  Interfaces to the states of distributed activities  E.g. workflow, video conf., dist. data analysis, subscription  Significant implications for how services are managed, named, discovered, and used  In fact, much of Grid is concerned with the management of service instances
  49. 49.  Naming and bindings  Every service instance has a unique name, from which can discover supported bindings  Lifecycle  Service instances created by factories  Destroyed explicitly or via soft state  Information model  Service data associated with Grid service instances, operations for accessing this info  Basis for service introspection, monitoring, discovery  Notification  Interfaces for registering existence, and delivering notifications of changes to service data
  50. 50.  Refactor Globus protocol suite to enable common base and expose key capabilities  Service orientation to virtualize resources and unify resources/services/information  Embrace key Web services technologies for standard IDL, leverage commercial efforts  Result: standard interfaces & behaviors for distributed system management: the Grid service
  51. 51.  Objectives of OGSA  Manage resources across distributed heterogeneous platforms  Deliver seamless quality of service  Provide a common base for autonomic management solutions  Define open, published interfaces  Exploit industry standard integration technologies
  52. 52.  Three principal elements of OGSA: OGSI, OGSA services, OGSA applications OGSA main architecture
  53. 53.  OGSI: provides a uniform way for software developers to model and interact with grid services by providing interfaces for discovery, life cycle, state management, creation and destruction, event notification, and reference management.
  54. 54. service data, keep-alive, notifications, service invocation create service grid service handle service requester service discovery service registry register service resource allocation service factory service instances
  55. 55.  OGSI and web services: OGSA architecture enhances Web services to accommodate requirements of the grid. These enhancements are specified in OGSI. Over time, it's expected that much of the OGSI functionality will be incorporated in Web services standards.
  56. 56.  Implementation of OGSI: The Globus Toolkit 3 (GT3) is the first full-scale implementation of the OGSI standard.
  57. 57.  OGSA architected services
  58. 58.  Grid core services
  59. 59.  Grid program execution and data services
  60. 60.  Grid program execution and data services hosting
  61. 61.  Service orientation to virtualize resources  Everything is a service  From Web services  Standard interface definition mechanisms  Evolving set of other standards: security, etc.  From Grids (Globus Toolkit)  Service semantics, reliability & security models  Lifecycle management, discovery, other services  A framework for the definition & management of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002
  62. 62. Service data element Service data element Service data element GridService … other interfaces … Implementation Service data access Explicit destruction Soft-state lifetime Notification Authorization Service creation Service registry Manageability Concurrency Reliable invocation Authentication Hosting environment/runtime (“C”, J2EE, .NET, …)
  63. 63. OnlineStore Persistent GS queryItems() StockMgr Persistent GS getItem() StockItemAF45X Transient GS Grid service client Item SDEs: dimension, price 3. create Factory 2. Get reference 1. query 4. query
  64. 64. 1) WSDL conventions and extensions for describing and structuring services  Useful independent of “Grid” computing 2) Standard WSDL interfaces & behaviors for core service activities  portTypes and operations => protocols
  65. 65.  portType (standard WSDL)  Define an interface: a set of related operations  serviceType (extensibility element)  List of port types: enables aggregation  serviceImplementation (extensibility element)  Represents actual code  service (standard WSDL)  instanceOf extension: map descr.->instance  compatibilityAssertion (extensibility element)  portType, serviceType, serviceImplementation
  66. 66.  Introduction to Globus Project  API and Protocols  Layered Architecture  Web services  Open Grid Service Architecture
  67. 67. Session - 3
  68. 68.  OGSI specification  Standard Interface  Service data  Grid service  Example Grid Service  Summary
  69. 69.  Defines WSDL conventions and extensions  For describing and naming services  Working with W3C WSDL working group to drive OGSI extensions into WSDL 1.2  Defines fundamental interfaces (using extended WSDL) and behaviors that define a Grid Service  A unifying framework for interoperability & establishment of total system properties  http://www.ggf.org/ogsi-wg
  70. 70.  OGSI defines basic patterns of interaction, which can be combined with each other and with custom patterns in a myriad of ways  OGSI Specification focuses on:  Atomic, composable patterns in the form of portTypes/interfaces  Define operations & associated service data elements  A model for how these are composed  Compatible with WSDL 1.2  Complete service descriptions are left to other groups that are defining real services
  71. 71. service PortTypePortType service service service Standard WSDL … … … Service Description Service Instantiation PortType serviceImplementation serviceImplementation … = serviceType serviceType … cA c A cA compatibilityAssertion= c A instanceOf instanceOf instanceOf instanceOf
  72. 72.  Naming and bindings  Every service instance has a unique name, from which can discover supported bindings  Information model  Service data associated with Grid service instances, operations for accessing this info  Lifecycle  Service instances created by factories  Destroyed explicitly or via soft state  Notification  Interfaces for registering interest and delivering notifications
  73. 73.  A Grid service instance maintains a set of service data elements  XML fragments encapsulated in standard <name, type, TTL- info> containers  Includes basic introspection information, interface-specific data, and application data  FindServiceData operation (GridService interface) queries this information  Extensible query language support  See also notification interfaces  Allows notification of service existence and changes in service data
  74. 74. • A DBaccess Grid service will support at least two portTypes – GridService – DBaccess • Each has service data – GridService: basic introspection information, lifetime, … – DBaccess: database type, query languages supported, current load, …, … Grid Service DBaccess DB info Name, lifetime, etc.
  75. 75.  Every service instance has a unique and immutable name: Grid Service Handle (GSH)  Basically just a URL  Handle must be converted to a Grid Service Reference (GSR) to use service  Includes binding information; may expire  Separation of name from implementation facilitates service evolution  The HandleMap interface allows a client to map from a GSH to a GSR  Each service instance has home HandleMap
  76. 76.  The Registry interface may be used to register Grid service instances with a registry  A set of Grid services can periodically register their GSHs into a registry service, to allow for discovery of services in that set  Registrations maintained in a service data element associated with Registry interface  Standard discovery mechanisms can then be used to discover registered services  Returns a WS-Inspection document containing the GSHs of a set of Grid services
  77. 77.  GS instances created by factory or manually; destroyed explicitly or via soft state  Negotiation of initial lifetime with a factory (=service supporting Factory interface)  GridService interface supports  Destroy operation for explicit destruction  SetTerminationTime operation for keepalive  Soft state lifetime management avoids  Explicit client teardown of complex state  Resource “leaks” in hosting environments
  78. 78.  Factory interface’s CreateService operation creates a new Grid service instance  Reliable creation (once-and-only-once)  CreateService operation can be extended to accept service-specific creation parameters  Returns a Grid Service Handle (GSH)  A globally unique URL  Uniquely identifies the instance for all time  Based on name of a home handleMap service
  79. 79. Grid Service DBaccess DB info Name, lifetime, etc. Grid Service DBaccess Factory Factory info Instance name, etc. Grid Service Registry Registry info Instance name, etc. Grid Service DBaccess DB info Name, lifetime, etc. “What services can you create?” “What database services exist?” “Create a database service”
  80. 80. User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider “I want to create a personal database containing data on e.coli metabolism” . . . Database Factory
  81. 81. User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . “Find me a data mining service, and somewhere to store data” Database Factory
  82. 82. User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . GSHs for Mining and Database factories Database Factory
  83. 83. User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . “Create a data mining service with initial lifetime 10” “Create a database with initial lifetime 1000” Database Factory
  84. 84. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Miner “Create a data mining service with initial lifetime 10” “Create a database with initial lifetime 1000”
  85. 85. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Miner Query Query
  86. 86. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Miner Query Query Keepalive Keepalive
  87. 87. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Miner Keepalive Keepalive Results Results
  88. 88. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Miner Keepalive
  89. 89. User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service . . . Compute Service Provider . . . Database Keepalive
  90. 90.  NotificationSource for client subscription  One or more notification generators  Generates notification message of a specific type  Typed interest statements: E.g., Filters, topics, …  Supports messaging services, 3rd party filter services, …  Soft state subscription to a generator  NotificationSink for asynchronous delivery of notification messages  A wide variety of uses are possible  E.g. Dynamic discovery/registry services, monitoring, application error notification, …
  91. 91.  Notifications can be associated with any (authorized) service data elements Grid Service DBaccess DB info Name, lifetime, etc. Grid Service DB info Name, lifetime, etc. Notification Source Notification Sink Subscribers
  92. 92.  Notifications can be associated with any (authorized) service data elements Grid Service DBaccess DB info Name, lifetime, etc. Grid Service DB info Name, lifetime, etc. Notification Source “Notify me of new data about membrane proteins” Subscribers Notification Sink
  93. 93.  Notifications can be associated with any (authorized) service data elements Grid Service DBaccess DB info Name, lifetime, etc. Grid Service DB info Name, lifetime, etc. Notification Source Keepalive Notification Sink Subscribers
  94. 94.  Notifications can be associated with any (authorized) service data elements Grid Service DBaccess DB info Name, lifetime, etc. Grid Service Notification Sink DB info Name, lifetime, etc. Notification Source New data Subscribers
  95. 95.  Service orientation to virtualize resources  Everything is a service  From Web services  Standard interface definition mechanisms: multiple protocol bindings, local/remote transparency  From Grids  Service semantics, reliability and security models  Lifecycle management, discovery, other services  Multiple “hosting environments”  C, J2EE, .NET, …
  96. 96.  Defines WSDL conventions and extensions  For describing and naming services  Working with W3C WSDL working group to drive OGSI extensions into WSDL 1.2  Defines fundamental interfaces (using extended WSDL) and behaviors that define a Grid Service  A unifying framework for interoperability & establishment of total system properties  http://www.ggf.org/ogsi-wg
  97. 97.  OGSI defines basic patterns of interaction, which can be combined with each other and with custom patterns in a myriad of ways  OGSI Specification focuses on:  Atomic, composable patterns in the form of portTypes/interfaces  Define operations & associated service data elements  A model for how these are composed  Compatible with WSDL 1.2  Complete service descriptions are left to other groups that are defining real services
  98. 98.  Naming and bindings (basis for virtualization)  Every service instance has a unique name, from which can discover supported bindings  Lifecycle (basis for fault resilient state management)  Service instances created by factories  Destroyed explicitly or via soft state  Information model (basis for monitoring & discovery)  Service data (attributes) associated with GS instances  Operations for querying and setting this info  Asynchronous notification of changes to service date  Service Groups (basis for registries & collective svcs)  Group membership rules & membership management  Base Fault type 110
  99. 99.  Attributes: Publicly visible state of the service  Want to bring full power of XML to attributes  getXXX/setXXX is too limiting  How to get/set multiple?  Want richer queries across attributes (e.g., join)  Use XML Schema, XPath, XQuery, XSLT, etc.  OGSI service data:  Attributes defined using XML Schema  Attributes combined into a single (logical) document within the service  Rich pull/push/set operations against service data document  Should declare attributes in WSDL interface 111
  100. 100. 112 Implementation Service data element Other standard interfaces: factory, notification, collections Hosting environment/runtime (“C”, J2EE, .NET, …) Service data element Service data element GridService (required) Data access Lifetime management • Explicit destruction • Soft-state lifetime Introspection: • What port types? • What policy? • What state? Client Grid Service Handle Grid Service Reference handle resolution
  101. 101. 113 Service registry Service requestor (e.g. user application) Service factory Create Service Grid Service Handle Resource allocation Service instances Register Service Service discovery Interactions standardized using WSDL Service data Keep-alives Notifications Service invocation Authentication & authorization are applied to all requests
  102. 102.  OGSI requires interface extension/composition  W3C WSDL working group defined standard interface extension in WSDL 1.2 that meets OGSI requirements  But could not wait for WSDL 1.2  So defined gwsdl:portType that extends WSDL 1.1 portType with:  WSDL 1.2 portType extension  WSDL 1.2 open content model  Define GWSDL  WSDL 1.1 & 1.2 mappings 114
  103. 103.  Globus Toolkit version 3.0 (Java, C client)  U Virginia OGSI.NET (.NET)  LBNL pyGlobus (Python)  U Edinburgh (.NET)  U Manchester (PERL)  Fujitsu Unicore (Java) 115
  104. 104. 116 <wsdl:definitions> <wsdl:types>…</wsdl:types> <wsdl:message>…</wsdl:message> … <gwsdl:portType name=“foo” extends=“ns:bar ogsi:GridService”> <wsdl:operation name=“op1”>…</wsdl:operation> <wsdl:operation name=“op2”>…</wsdl:operation> <ogsi:serviceData … /> </gwsdl:portType> … </wsdl:definitions>
  105. 105. 117 Web Services: Basic Functionality OGSA OGSI: Interface to Grid Infrastructure Applications in Problem Domain X Compute, Data & Storage Resources Distributed Application & Integration Technology for Problem Domain X Users in Problem Domain X Virtual Integration Architecture Generic Virtual Service Access and Integration Layer - Structured Data Integration Structured Data Access Structured Data Relational XML Semi-structured Transformation Registry Job Submission Data Transport Resource Usage Banking Brokering Workflow Authorisation
  106. 106. 118 1a. Request to Registry for sources of data about “x” 1b. Registry responds with Factory handle 2a. Request to Factory for access to database 2c. Factory returns handle of GDS to client 3a. Client queries GDS with XPath, SQL, etc 3b. GDS interacts with database 3c. Results of query returned to client as XML SOAP/HTTP service creation API interactions Registry Factory 2b. Factory creates GridDataService to manage access Grid Data Service Client XML / Relational database Slide Courtesy Malcolm Atkinson, UK eScience Center
  107. 107.  OGSI specification  Standard Interface  Service data  Grid service  Example Grid Service
  108. 108. Session - 4
  109. 109.  Introduction  GT3 Core Services  Base Services  User-Defined Services  Writing a Simple Grid Service  Summary
  110. 110.  Open source implementation of OGSI 1.0  Usable Web Services – based grid service toolkit  Runtime environment called the container  Written in Java
  111. 111.  Implements all OGSI interfaces  GridService, Factory, Notification, HandleResolver  Admin, Logging, Management …  Runtime environment for services  Between application and plumbing  Embedded: library for any J2SE application  Standalone: lightweight server  Web: runs in a J2EE server  EJB: exposes stateful Entity and Session beans as services
  112. 112.  Transport and message-level security  Transport being deprecated  Message based on WS-Security, XML Signature  At SOAP level  Per-session or per message  SSL, X.509 certificates  Based on JAAS  Authentication, Authorization Service Framework
  113. 113.  Database services  Persisting Service Data in native XML db  Managed Job Service  Notify, Subscribe, Pause, Stop, Query jobs  Index Service  Query service around grid  Reliable File Transfer  Guaranteed large file transfers via restart  Replica Location Service
  114. 114. 126 GT2 Components RLS GT-OGSA Grid Service Infrastructure The focus of this presentation
  115. 115. 127 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  116. 116. 128 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  117. 117.  GT3 includes a set of primitives that fully implement the interfaces and behaviors defined in the OGSI Specification  Defines how entities can create, discover and interact with a Grid service  The OGSI Specification defines a protocol: GT3 provides a programming model for that protocol 129
  118. 118.  Defines the fundamental behavior of a Grid Service  Introspection  Discovery  Soft State Lifetime Management  Mandated by the Spec
  119. 119.  destroy()  setServiceData()  findServiceData()  requestTerminationAfter()  requestTerminationBefore()  addOperationProvider() (See docs for complete set of methods) 131 Implementation of the OGSI Spec <parameter name=“operationProviders” value=“<className>”> Additional functionality can be added to a Grid Service using OperationProviders Deployment descriptor
  120. 120. 132 Write service-specific logic that also implements the GT3 OperationProvider interface
  121. 121. 133 Write service-specific logic that also implements the GT3 OperationProvider interface Combine with one of the two GT3 implementations of base GridService functionality: GridServiceImpl or PersistentGridServiceImpl
  122. 122. 134 Write service-specific logic that also implements the GT3 OperationProvider interface Combine with one of the two GT3 implementations of base GridService functionality: GridServiceImpl or PersistentGridServiceImpl An OGSI-Compliant grid service
  123. 123. 135 An OGSI-Compliant grid service OperationProviders are configured at deployment time or added at runtime Write service-specific logic that also implements the GT3 OperationProvider interface Combine with one of the two GT3 implementations of base GridService functionality: GridServiceImpl or PersistentGridServiceImpl
  124. 124.  OPs can be designed as atomic bits of functionality to facilitate reuse  OP approach eases the task of bringing legacy code into OGSI-compliance  OPs allow Grid Services to be formed dynamically (in contrast to the inheritance approach) 136
  125. 125. 137 NotificationSourceProvider HandleResolverProvider ServiceGroupRegistrationProvider ServiceGroupProvider FactoryProvider
  126. 126.  Our NotificationSourceProvider implementation allows any Grid Service to become a sender of notification messages  A subscribe request on a NotificationSource triggers the creation of a NotificationSubscription service  A NotificationSink can receive notification msgs from NotificationSources  Sinks are not required to implement the GridService portType  Notifications can be set on SDEs 138
  127. 127. Factory portType  Factories create services  Factories are typically persistent services  Factory is an optional OGSI interface (Grid Services can also be instantiated by other mechanisms) 139
  128. 128. Service group portTypes  A ServiceGroup is a grid service that maintains information about a group of other grid services  The classic registry model can be implemented with the ServiceGroup portTypes  A grid service can belong to more than one ServiceGroup  Members of a ServiceGroup can be heterogeneous or homogenous  Each entry in a service group can be represented as its own service  Service group portTypes are optional OGSI interfaces 140
  129. 129. HandleResolver portType  Defines a means for resolving a GSH (Grid Service Handle) to a GSR (Grid Service Reference)  A GSH points to a Grid Service (GT3 uses a hostname-based GSH scheme)  A GSR specifies how to communicate with the Grid Service (GT3 currently supports SOAP over HTTP, so GSRs are in WSDL format) 141 Includes 0 or more Grid Service Handles (GSHs) Includes 0 or more Grid Service References (GSRs) Service locator
  130. 130. 142 Registry * All scenarios are offered as examples and are not prescriptive 1. From a known registry, the client discovers a factory by querying the service data of the registry Client
  131. 131. 143 Client Registry 2. The client calls the createService operation on the factory Factory 1. From a known registry, the client discovers a factory by querying the service data of the registry
  132. 132. 144 Client Registry Factory Service 3. The factory creates a service 1. From a known registry, the client discovers a factory by querying the service data of the registry 2. The client calls the createService operation on the factory
  133. 133. 145 Client Registry Factory Service 4. The factory returns a locator to the newly created service 1. From a known registry, the client discovers a factory by querying the service data of the registry 2. The client calls the createService operation on the factory 3. The factory creates a service
  134. 134. 146 Client Registry Factory Service 5. The client and service interact 1. From a known registry, the client discovers a factory by querying the service data of the registry 2. The client calls the createService operation on the factory 3. The factory creates a service 4. The factory returns a locator to the newly created service
  135. 135. 147 NotificationSi nk 1. NotificationSink calls the subscribe operation on NotificationSource NotificationSo urce
  136. 136. 148 2.NotificationSou rce creates a subscription service 1. NotificationSink calls the subscribe operation on NotificationSource NotificationSi nk NotificationSo urce Notification Subscription
  137. 137. 149 3. Notification Source returns a locator to the subscription service 1. NotificationSink calls the subscribe operation on NotificationSource 2.NotificationSou rce creates a subscription service NotificationSi nk NotificationSo urce Notification Subscription
  138. 138. 150 Notification Subscription4.b The NotificationSink and Subscription service interact to perform lifetime management 4.a deliverNotification stream continues for the lifetime of NotificationSubscription 1. NotificationSink calls the subscribe operation on NotificationSource 2.NotificationSou rce creates a subscription service 3. Notification Source returns a locator to the subscription service NotificationSi nk NotificationSo urce
  139. 139. 151 NotificationS ink NotificationS ource NotificationS ubscription 1. NotificationSink calls the subscribe operation on NotificationSource subscribe 2.NotificationSou rce creates a subscription service 3. Notification Source returns a locator to the subscription service 4.b The NotificationSink and Subscription service interact to perform lifetime management 4.a deliverNotification stream continues for the lifetime of NotificationSubscription The sole mandated cardinality: 1 to 1
  140. 140. 152 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  141. 141.  Transport Layer Security/Secure Socket Layer (TLS/SSL)  To be deprecated  SOAP Layer Security  Based on WS-Security, XML Encryption, XML Signature  GT3 uses X.509 identity certificates for authentication  It also uses X.509 Proxy certificates to support delegation and single sign-on, updated to conform to latest IETF/GGF draft 153
  142. 142. 154 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  143. 143.  General-purpose services that facilitate the use of Grid services in production environments  The 3.0 distribution includes three system-level services  An administration service  A logging service  A management service 155
  144. 144. 156 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  145. 145. Includes the OGSI Implementation, security infrastructure and system-level services, plus:  Service activation, deactivation, construction, destruction, etc.  Service data element placeholders that allow you to fetch service data values dynamically, at query time  Evaluator framework (supporting ByXPath and ByName notifications and queries)  Interceptor/callback framework (allows one to intercept certain service lifecycle events) 157
  146. 146. 158 Interface Layer Transport Layer Implementation Layer Layers in the Web Services Model OGSI Spec is here Transport/binding layer (e.g., GT3: SOAP over HTTP) Container is here
  147. 147. 159 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  148. 148. GT3 currently offers support for four Java hosting environments:  Embedded  Standalone  Servlet  EJB 160
  149. 149.  Virtual hosting allows Grid services to be distributed across several remote containers  Useful in implementing solutions for problems common to distributed computing  Load balancing  User account sandboxing 161
  150. 150. 162 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  151. 151.  GRAM Architecture rendered in OGSA  The MMJFS runs as an unprivileged user, with a small highly-constrained setuid executable behind it  Individual user environments are created using virtual hosting 163 MMJFS MJS MJS MJS MJS User 1 User 2 User 3 Master User MJS MJS MMJFS: Master Managed Job Factory Service MJS: Managed Job Service User Hosting Environment
  152. 152. 164 Client Index Service 1. From an index service, the client chooses an MMJFS
  153. 153. 165 Client Index Service 2. The client calls the createService operation on the factory and supplies RSL MMJFS 1. From an index service, the client chooses an MMJFS
  154. 154. 166 Client Index Service MMJFS MJS 3. The factory creates a Managed Job Service 1. From an index service, the client chooses an MMJFS 2. The client calls the createService operation on the factory and supplies RSL
  155. 155. 167 Client Index Service MMJFS MJS 4. The factory returns a locator 1. From an index service, the client chooses an MMJFS 2. The client calls the createService operation on the factory and supplies RSL 3. The factory creates a Managed Job Service
  156. 156. 168 Client Index Service MMJFS MJS 5. The client subscribes to the MJS’ status SDE and retrieves output 1. From an index service, the client chooses an MMJFS 2. The client calls the createService operation on the factory and supplies RSL 3. The factory creates a Managed Job Service 4. The factory returns a locator
  157. 157.  Index service as caching aggregator  Caches service data from other Grid services  Index service as provider framework  Serves as a host for service data providers that live outside of a Grid service to publish data 169
  158. 158.  OGSI-compliant service exposing GridFTP control channel functionality  3rd-party transfer between GridFTP servers  Recoverable Grid service  Automatically restarts interrupted transfers from the last checkpoint  Progress and restart monitoring 170 GridFTP Server 1 GridFTP Server 2 RFT JDBC
  159. 159. 171 Performance Policy Faults service data elements Pending File Transfer Internal State Grid Service Notf’n Source Policy interfacesQuery &/or subscribe to service data Fault Monitor Perf. Monitor Client Client Client Request and manage file transfer operations Data transfer operations
  160. 160. 172 GT-OGSA Grid Service Infrastructure Security Infrastructure System-Level Services Base Services User-Defined Services Grid Service Container Hosting Environment Web Service Engine OGSI Spec Implementation
  161. 161.  GT3 can be viewed as a Grid service development kit that includes:  Primitives, APIs, tools and runtime services designed to ease the task of building OGSI- compliant services  Primitives for provisioning security  Base services that provide an infrastructure with which to build higher-level services 173
  162. 162. 174 ANT User source files GT3 build files User build file Grid service executable files (Diagram inspired by Borja Sotomayor’s excellent GT3 tutorial)
  163. 163. OGSA OGSI GRID SERVIC ES Web Services GT3 Implements
  164. 164.  Define all aspects of the service interface  Java Interface and / or GWSDL  Generate grid service support code  Server and client stub classes  Implement the service guts  Can be automated using ant
  165. 165.  Message of the day service  Difference from normal WebService  Stateful server keeps track of which message sent  Messages delivered in order  Will not repeat until MOTDs delivered  Client calls instance’s getMOTD()
  166. 166.  Bottom-Up  Write a Java Interface  Run a tool against it to generate WSDL  Some complex Java data types don't map well into WSDL  Good for exposing legacy code w/o reprogramming  Top-Down  Write a GWSDL Port Type definition  Run two tools against it to generate full WSDL
  167. 167.  Write the abstract definition of the service  Types, Messages and PortType in GWSDL  Better for more complex service data types  Applying XML syntax  Run through GWSDL2WSDL tool  Creates WSDL 1.1 port type definition  Run through the generateBindings tool  Create the wsdl:binding and wsdl:service parts of portType definition
  168. 168.  Generates Java stubs  Handles marshalling of data to / from XML  Server and client side JAX-RPC compliant interfaces  All done by Ant
  169. 169.  Write a Java class that does the job  Implement the service interface  Private attributes make it stateful  Private methods in the class are OK  Simple constructor  Calls super() to pass to GridServiceImpl  Implements service methods  String getMOTD()
  170. 170.  Write a deployment descriptor  Tells the Web server how the service is published  Extra fields define factory attributes  Create a GAR  Grid Archive, a JAR with grid specific extras  Best to copy/paste an existing ant task  target=“makeGar”  Deploy the GAR into a hosting environment  ant deploy –D gar.name = <path to GAR>
  171. 171.  Command-line client application  Good idea to pass service info on command line  Or use Index Service to discover  Make a GSH from the factory service URI  Build up with hostname and factory path  Locate the factory using the GSH  Create a service instance with the factory  Call a remote method on the instance  Destroy the instance
  172. 172.  Start the container  ant startContainer  If using factory, don’t need to create an instance  If not using a factory, create a service instance  java.org.globus.ogsa.client.CreateService <serverURL>/<factory service name>  ServerURL=http://hostname:8080  Factory = /<path to factory>  Container now running, waiting for requests
  173. 173.  Launch the client  Pass on the command line  Instance reference (GSR)  Like handle, but with /hash-123 … on end  Client pulls and displays one MTD
  174. 174.  Requirements  jakarta-tomcat-4.1.24 - web services container  ogsi-1.0 - GT3 implementation  ogsi - symbolic link to ogsi-1.0  Sample source code and tutorial available here  http://gdp.globus.org/gt3-tutorial/  Set the environment variables as required
  175. 175.  Math Web service can perform the following functions  Addition  Subtraction  Multiplication  Division
  176. 176.  PortType – defines operations  We can either  Write the WSDL directly (more flexible)  Generate WSDL from an interface language (easy)
  177. 177. package gt3tutorial.core.first.impl; public interface Math{ public int add(int a, int b); public int subtract(int a, int b); public int multiply(int a, int b); public float divide(int a, int b); } ${TUTORIAL_DIR}/gt3tutorial/core/first/impl /Math.java
  178. 178.  cd ${TUTORIAL_DIR}/gt3tutorial/core/first/impl  javac Math.java  cd ${TUTORIAL_DIR}  java org.apache.axis.wsdl.Java2WSDL -P MathPortType -S MathService -l http://localhost/ogsa/services/core/first/Math Service -n http://first.core.gt3tutorial/Math gt3tutorial.core.first.impl.Math  more MathService.wsdl
  179. 179.  -o indicates the name of the output WSDL file  -l indicates thelocation of the service  -n is the target namespace of the WSDL file  -p indicates a mapping from the package to a namespace. There may be multiple mappings.  the class specified contains the interface of the webservice.
  180. 180.  java org.globus.ogsa.tools.wsdl.DecorateWSDL $GLOBUS_DIRECTORY/schema/ogsi/ogsi_ bindings.wsdl MathService.wsdl  Opening service file: MathService.wsdl
  181. 181.  java org.globus.ogsa.tools.wsdl.GSDL2Java MathService.wsdl  Creates:  ${TUTORIAL_DIR}/gt3tutorial/core/first  ${TUTORIAL_DIR}/org
  182. 182.  cd $TUTORIAL_DIR/gt3tutorial/core/first/imp l package gt3tutorial.core.first.impl; import org.globus.ogsa.impl.ogsi.GridServiceImpl; import gt3tutorial.core.first.Math.MathPortType; import java.rmi.RemoteException; ${TUTORIAL_DIR}/gt3tutorial/core/first/impl/MathImpl.java
  183. 183. • Class definition public class MathImpl extends GridServiceImpl implements MathPortType { [constructor, add, divide, subtract, multiply] } Contains basic functionality common to every Grid Service; must implement this class Our autogenerated stub ${TUTORIAL_DIR}/gt3tutorial/core/fir st/impl/MathImpl.java
  184. 184. • Written in WSDD (Web Services Deployment Descriptor) <?xml version="1.0"?> <deployment name="defaultServerConfig”> <service name="tutorial/core/first/MathService"> <parameter name="allowedMethods" value="*"/> <parameter name="className" value="gt3tutorial.core.first.impl.MathImpl"/> </service> </deployment> ${TUTORIAL_DIR}/gt3tutorial/core/first/Math.wsdd Grid Service name http://localhost:8080/ogsa/services/tutorial/core/first/MathService. All methods public This class implements the service
  185. 185.  cd $TUTORIAL_DIR  Stubs  javac -sourcepath ./ gt3tutorial/core/first/Math/*.java  Implementation  javac -sourcepath ./ gt3tutorial/core/first/impl/*.java
  186. 186.  jar cvf Math-stub.jar gt3tutorial/core/first/Math/*.class  jar cvf Math.jar gt3tutorial/core/first/impl/*.class  mkdir -p gar/schema/tutorial  mv Math.jar Math-stub.jar gar  cp MathService.wsdl gar/schema/tutorial  cp gt3tutorial/core/first/Math.wsdd gar/server-deploy.wsdd  jar cvf Math.gar -C gar .  Result is Math.gar
  187. 187.  To deploy services in tomcat  Step 1: deploy service in the gt3 container (run this command from ${GLOBUS_LOCATION})  ant deploy -Dgar.name=${TUTORIAL_DIR}/Math.gar  Step 2: deploy gt3 container services to tomcat (run this command from your ogsi directory)  ant -Dtomcat.dir=${CATALINA_HOME} deployTomcat  (extra) cp server-config.wsdd ${CATALINA_HOME}/webapps/ogsa/WEB-INF  Step 3: Restart tomcat  ${CATALINA_HOME}/bin/tomcat -s start  Test: http://<machine name>:<port>/ogsa/services
  188. 188. int a = Integer.parseInt(args[1]); int b = Integer.parseInt(args[2]); URL GSH = new java.net.URL(args[0]); // Get a reference to the remote web service MathServiceLocator mathService = new MathServiceLocator(); MathPortType math = mathService.getMathService(GSH); // Call remote method 'add’ int sum = math.add(a,b); // Print result System.out.println(a + " + " + b + " = " + sum); ${TUTORIAL_DIR}/gt3tutorial/core/first/client/MathClient.java
  189. 189.  cd $TUTORIAL_DIR  javac -sourcepath ./ gt3tutorial/core/first/client/*.ja va
  190. 190.  java gt3tutorial.core.first.client.Math Client http://localhost:<port>/ogsa/servi ces/tutorial/core/first/MathServic e 3 2
  191. 191.  Introduction  GT3 Core Services  Base Services  User-Defined Services  Writing a Simple Grid Service

×