The organization had been using Oracle RDBMS, Oracle Designer & Forms and even an Oracle EBS module for many years. On the side it had been running several open source J2EE web applications. Facing several new challenges, it took the plunge into SOA - the technology and the architectural principle. This presentation tells their story. It started with the business need of opening up the core application to several external business partners. A programmatic interface was required for submitting expense reports - in the thousands - for one business partner, who also wanted to be able to ask for the status for each one those reports. Another external entity needed the ability to learn about relevant changes in product and pricing data through an API. We will discuss how SOA principles were used to design the application architecture. And how the Oracle 10g SOA Suite - specifically ESB and BPEL PM - were used to implement the requirements. We go into the choices the organization had to make, the challenges they had to overcome, the skills they had to acquire and the results they achieved. After this first stage came the next set of business requirements needed tackling. And now the first benefits could be reaped. Following the guidelines established in their first close encounter with SOA, this organization achieved the first reuse of services, could rapidly decide on the application architecture for the ADF 11g Internet Application that needed to be created and further expanded their still little SOA universe. The initial experience now enabled them to decide on whether and how to service enable specific functionality required for the web application - how to use ESB and BPEL, for example and when to use application specific database APIs rather than SOA Web Services. This stage also taught them the necessity of Governance - what are naming conventions for elements in Schema Definitions and Services, who owns a service, what’s the required availability and how is that achieved, what are the SLAs (Service Level Agreements) around the service, how can the service be evolved with respect to new or changing needs. The presentation will tell the story of the two stages and how the organization went about them. It will show some small demos to illustrate what was done. It will share some conclusions as to what works and what does not. Finally it briefly discusses the future plans for this organization with regard to SOA. The presentation is for an audience that probably (though not necessarily) has a classic Oracle background and either is in the process of taking its first steps in the SOA arena or considers moving their. It should help make that process more tangible and hopefully realistic and desirable. Summary: The organization had been using Oracle RDBMS, Oracle Designer & Forms and even an Oracle EBS module for many years. Facing several new challenges, it took the plunge into SOA - the technology and the architectural principle. This presentation tells their story. Of getting started with BPEL and ESB, with Governance and Security (OWSM) and of applying SOA principles. And of the second phase where reuse and agility started to occur.

1. The Story of How an Oracle Classic Stronghold Successfully Embraced SOA ODTUG Kaleidoscope 2009 Lucas Jellema SOA

2. Overview What is an Oracle stronghold? Triggers to start moving towards Services Levels of embracing Services and SOA Objectives, benefits, costs & challenges Demonstration Pitfalls, Lessons Learned & Best Practices Summary

3. What is an Oracle stronghold? The typical Oracle stronghold Using Oracle RDBMS & Oracle Development tools Lot of SQL and PL/SQL Probably Oracle Forms and maybe APEX as well Possibly Oracle Designer, tools for BI & Reporting Several databases with many years of essential corporate data IT staff has Oracle veterans – 5-15 years or more Internet development may have taken place largely separate from the Oracle technology stack

4. My central case… Oracle Forms application tightly integrated with an Oracle RDBMS The database is known by the name of the app Containing jobs, workers, timesheets, payments,.. Used in hundreds of branches as well as the central (back)office Main business driver that required attention Business partners requested a programmatic interface to load multiple timesheets – to save on time and hassle (time again)

5. Similar cases … Oracle Forms application for assigning homes offered SaaS-style SaaS Customers want the details on houses and their availability published on a website .NET applications need access to data in the Oracle Database Customers want their local applications to interface (programmatically) with the SaaS application

6. Then wat happened… (2) Car Lease company has various custom applications and databases per department Business requires IT to support processes that go across those applications and databases through a single, unified User Interface involving a legacy database and a 3rd party ERP system with eventually some self service web modules Insurance company sells policies through agents using a Forms application New direct channel: On-line policy selling

7. Then wat happened… (3) Agricultural company supports ‘cow insemination’ process with Forms application Farmers and inspectors need to be able to record data anytime and anywhere through PDAs (that run a .Net application) Mid-sized chemical pharmaceutical company uses BoB stand-alone systems and databases To allow for faster (near real-time) responses to customer demands and logistical challenges, tighter integration between the systems is needed

8. Common Characteristics Cross Boundaries Cross Technology - .Net, Uniface, Java, Tibco Cross Channel – Back Office, Web, PDA, API Cross User Group – Internal, Agents, Self Service Cross Domain – Multiple departments & systems Cross Enterprise – Interact with external partners Data synchronization or consolidation Management whim Vision From Database angle: providing services

9. Objectives Business Agility (Faster) responses to changing demands Or at least an urgent, currently pending demand Creating new business from existing resources IT Flexibility Optimize locally without impact ‘globally’ Prepare for future developments Lower costs Through reuse, better integration, decoupling

10. Objectives (longer term) Higher Quality and Faster Process execution Automated data exchange cross boundaries Workflow and task orientation integrated, cross department, in a controlled way based on sound understanding of the business processes Business Event driven interaction Manage risks and fear

11. SOA = BAD

12. Business Agility through Decoupling SOA =

13. Decoupling≈Managing Dependencies minimize impact of change while maximizing reusability

14. Types of decoupling Functional Interface - Encapsulation of implementation Design by Contract, Implement by Design Technical No proprietary technology, protocol, message format Standards based (XML, HTTP, RSS, WSDL…) Temporal Asynchronous communication (separate response) Development Separate teams working in parallel based on mutually agreed interface definitions

15. Decoupling Applications & Data Application(User Interface) Application(User Interface) Data Data

16. Decoupling Applications & Data Application(User Interface) Application(User Interface) Data

17. Decoupling Applications & Data Application (User Interface) Data

18. Decoupling Applications & Data Application (User Interface) WorkflowEngine CMS Email IM Fax

19. Data Ownership Data no longer exclusively owned by a single application Data (query and manipulation) available via APIs, (web)services and open standards For example based on XML, XSD, WSDL, SOAP, HTTP Data Hubs are formalized, structured approach where data is completely separated from applications All access is through services No data duplication at all in the enterprise Data ownership is separate process

20. Decoupling from Table to ESB+ http WEBDAV FTP WSRP http WS/SOAP WS* WS* WS*

21. Tables in Database SQL for retrieval and manipulation Data Model in plain “view” Decoupling between DML and Retrieval Retrieve data: select e.empno, e.ename, d.dnamefrom emp join dept using (deptno) Manipulate data: insert into dept;insert into emp;

22. View in Database Hide (encapsulate) Data Model Manage access privileges SQL for retrieval and manipulation Instead Of trigger decouples DML operations select id, name, department from emp_vw Insert into emp_vw Use case: new UI on top of ‘legacy’ data model

23. Package in Database Hide (encapsulate) SQL Procedure calls for retrieval and manipulation Potentially complex data structures using Object Types and (nested) Cursors HRM_MGR.get_emp( id) return emp_t HRM_MGR.create_emp( emp_t); Use case: tailor made business services to support clients that understand types Or core service implementation around which a non-type wrapper is applied

24. Database Object Types

25. Package in Database (2) Hide (encapsulate) SQL and Oracleand user defined Types Procedure calls for retrieval and manipulation Input and output parameters standard types only (string and number) Complex datastructures: XML passed as string HRM_MGR.get_emp( id) return string HRM_MGR.create_emp( string); Use case: packaged business services to support any client (that can access the DB)

26. Handling XML Structured, multi-level data in a string: XML Oracle Database has XMLType Can be created from a String, will parse XML Can validate against a schema definition (the XML data design) Support XPath operations to retrieve specific bits and pieces from the XML document Can do XSLT transformations of the incoming or outgoing XML

27. XML based PL/SQL interface

28. AQ for Asynchronous Decouple consumer and providerin time – asynchronuous processing Consumer is registered on the AQ Usually a package that processes User Defined Type that is sent as payload in the AQ message Use case: asynchronously processed one-way (fire-n-forget) requests Potentially lenghty requests High volume of requests

29. Hiding the database http WEBDAV FTP http WS/SOAP WS* WS* WS*

30. Publish package as http-based API using dbms_epg http Hide database protocol Not its physical location nor the schema, and user authentication HTTP communication is truly cross technology Browser, Java, .Net, JavaScript & RIA clients, … Approximation of RESTful services (very du jour) Can publish in various formats Text, HTML, CSV, JSON, XML, RSS Use case:cross-technology, internal no WS*/ESB

31. HTTP API – directly on top of the RDBMS

32. Publish static resources on various protocols with XMLDB Hide database protocol support FTP, WEBDAV, HTTP(S) Run scheduled batch jobs (PL/SQL) to periodically create & expose resources Can also consume and process resources Use case: cross technology need for retrieving slowly changing resources (CSV, XML) Possibly uploading resources for batch processing Use case: http WEBDAV FTP

33. Publishing RDBMS resources in a decoupled fashion

34. SOAP based WebServices http WEBDAV FTP WSRP http WS/SOAP WS* WS* WS*

35. SOAP WebServices All messages (input and output) are XML The message consists of two parts inside an envelope (a SOAP XML wrapper) The header with meta-data The body with the contents to be handled by or returned by the service The WebService Definition Language (WSDL) document describes the service An XML Schema Document (XSD) describes the structure of the XML messages XSD is like an ERD or Table Design

37. Limited control over WSDL & XSD

39. Enterprise Service Bus App 2 External Partner Service Service Service Service Service App 1 Data Data

40. The Enterprise Service Bus Should first of all be considered ‘a pattern’ Virtualizes services – hides the real service from consumers Deals with the physical location of the services Adapts synchronous to a-synchronous and vv. Can use multiple real services to offer one virtual (composite) service Allows callers to use a generic, canonical message structure that it will transform to the service contract It may even allow callers to use their own “lingo”

41. Canonical Data Model Common Business Language Esperanto for service invokers Service Service Data Data Clients Customers

42. The Enterprise Service Bus (2) Handles various QoS & SLA aspects sometimes in concert with tools like OWSM Encryption, signing, authentication Retry and fallback “Throttle” (prevent peak loads) Does monitoring, tracking & auditing, reporting, notification and escalation Works with Adapters to access technologies like RDBMS (SQL, PL/SQL), AQ and JMS, File System, FTP, Java, E-Business Suite

43. Publish PL/SQL Package through ESB WS* Use Database Adapter to createService, combine with ESB Routing Service Use case: external access to services virtualize location of service – route to service based on content of the request virtualize part of contract of service Package-derived XSD not suitable for consumers handle peak loads monitor service levels and trace service access

44. Enterprise Service Busin action One team responsible for exposing services to external consumers Working with those customers for establishing the contract and testing across the firewall This team built from internally provided services Team two worked inside database – providing package based API for granular services Team three created ESB level database adapter and routing/mapping services WS*

45. Evaluation – end of phase 1 Done well Working “application” in production! Worked together (across teams and departments) on the Service definition and the ‘XSD’ To determine the scope and granularity of services and operations for optimal reuse To define Management sponsorship (though IT mainly) To be improved More involvement from the business for defining both services and especially the canonical model

46. Evaluation – end of phase 1 Done well Achieved good way of working with database team developing packages underneath services Developed a feel for using services rather than immediate database access Built up XML and ESB skills To be improved Involvement of administrators Setting up Dev-Test-Acceptance-Production Managing Service End Points

47. Evaluation – end of phase 1 Done well Ambitions, drive - However: technology driven To be improved Define the Enterprise Architecture/BluePrint and work within its context “Think globally, act locally” Set up CEA – mandated by business and IT Reduce number of service calls Externally exposed services do not need to be 1:1 with database packages! Granularity and Composite Services

48. Evaluation – end of phase 1 To be improved Share success with business and other IT teams Establish a mechanism for sharing, exposing, finding, adopting SOA artefacts Services, Canonical Data Model, Policies/SLAs, … And also for ‘governing’ the life cycle: how to change services already being used? (automated) Testing Monitoring service traffic (some focus on SLA) Peak load, response time/overhead, availability

49. End of Phase 1 Working service implementation and infrastructure – across departments End-to-end from backoffice database to customer “Foundation for innovation” Lost the fears, ready for the next step Basis for reuse and widening the scope Shimmering light at the end of the tunnel for the Forms application

50. Improve & Broaden the scope of the Canonical Data Model Involve business representatives for all of business Add business terminology and (some) business rules that help describe the model Build up translations of domain values and business object identies across ‘systems’ More accessible and consistent structure using namespaces (domains), naming conventions and guidelines (element vs. attribute, data types) Annotate the model and provide examples Eridicate database model legacy or technology specific elements

51. Pending “Research” How to make use of cache infrastructure to prevent services being called unnecessarily Yet guaranteeing non-stale data How do we process binary attachments through the layers of our architecture Do we have some low-hanging-fruit to pick in order to gain some additional benefits Design patterns, best practices, short-cuts, … Security and Policies – what levels, mechanism

52. Phase 2 A new user interface (internet website) on top of the back office database Partially covered by existing services and an existing Web Application and CMS Business is also looking for Web 2.0, Social/Community-style interaction in Portal Running process flows that potentially go from external workforce through branch to (multiple entities within the) back office How to move from peak load batch processing

53. Introducing BPEL for Service WS* BPEL adds to service Long running (stateful) ‘service instances’ Composite services that include Multiple service calls (including asynchronous) Exception handling including retry and compensation Human Task for manual steps & integration Rule Engine Process flow logic Use case: data request must be fulfilled by various services; DML impacts several systems and/or requires human approval

54. Publish Business Events Extremely Decoupled Architecture Any system – including database – reports events that may be interesting to other parties The Event backbone (could be the ESB) Defines Event Types (name, structure of payload) Registers Event Listeners (“please call me when the event occurs and send the details”) Receives events – instances of the predefined event type with payload and timestamp Propagate events to all registered listeners Without blocking the event producer

55. Event Driven Architecture (EDA) External Partner App 2 Service Service Service Service Service App 1 Data Data

56. Database publishing events WS* Table Trigger intercepts DML Checks for Business Events such as new employee Sends them to package EVENT_PRODUCER Package EVENT_PRODUCER sends events Via UTL_HTTP to a WebService Via AQ to a listener (ESB AQ Adapter) EventProducer EMP

57. Publishing Service with UI Instead of only publishing a programmatic service interface A service can be published with a User Interface; the service-with-UI is called: Portlet The standard approach: Portlet Container in Application Server exposes WSRP services for the portlets The Portlet produces (X)HTML and handles HTTP requests A Portal consumes the WSRP Portlets in a web page WSRP

58. Decoupling from Table to ESB+ http WEBDAV FTP WSRP http WS/SOAP WS* WS* WS*

59. Increasingly decoupled More hiding of the implementation More Formal Interface Contract Less (proprietary) technology & more standards for interacting Less exposure of (legacy) data model More support for asynchronous interaction More reuse potential Pervasive throughout enterprise More suitable for external consumption

60. Comes at a cost… More run time overhead Additional tiers XML serialization and deserialization More infrastructure Burden of Administration License Costs Hardware Broader skills palette – more stuff to master Harder to get started

61. Pitfalls Do SOA from the IT/technology side only Inconsistent, illegible, unstructured namespaces and (XML) canonical model Having the database (table and column names) shine through in the canonical data model And forgetting the database design wisdom Reusable SOA artefacts are not found, understood nor trusted Lack of Balance between reusability and usefulness (Fine grained vs. coarse grained)

62. Pitfalls The greedy clutches of enterprise architects Think globally, talk, (high level) design, draw & write, present, think, talk, …. (no real action) … versus the technology driven, ’think and act locally’ approach from the ‘developer squad Introducing new unmanaged dependencies Hard coded endpoints (service URLs) Calling external services without proper SLA or fallback option Using complex technology without proper skills

63. Pitfalls No DTAP process & environment And/or completely manual Involving DBAs/Administrators way too late Inappropriate use of the SOA infrastructure Web applications retrieving each individual record (or even field) through a separate service call Running Forms on top of the ESB! Sending debug and trace messages via the ESB “Package calls other package via ESB”

64. Useful Use Mock Service implementations during development and test Developers that depend on services not yet available easily get stuck Automated Functional and Performance Test of individual Services Automatic Service ‘ping’ utility Early detection of service unavailability

65. Summary Objective: agility through decoupling Managing dependencies Crossing boundaries – functional, technology, time Just do it! (well, “think big, do (small)”) Get started – at the right level for your situation Do not go off and buy BPEL just like that Even though it won’t be perfect the first time round – you will learn (only) through experience Do it explicitly, visibly and with all involved SOA

66. Q &A SOA