Digital Integration Hub - Maximise Your APIs
•Descargar como PPTX, PDF•
0 recomendaciones•339 vistas
Presentation as delivered to the Brisbane Azure User Group on 10 November 2021
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Digital Integration Hub - Maximise Your APIs
Similar a Digital Integration Hub - Maximise Your APIs (20)
Más de Daniel Toomey
Más de Daniel Toomey (20)
Último
Último (20)
Digital Integration Hub - Maximise Your APIs
- 1. DIGITAL INTEGRATION HUB: MAXIMISE YOUR APIS Dan Toomey @daniel2me
- 2. Who am I? • Senior Integration Specialist, Deloitte • Microsoft Azure MVP • MCSE, MCT, MCPD, MCTS BizTalk & Azure • Pluralsight Author • www.mindovermessaging.com • @daniel2me
- 3. 3 "APIS MAKE DIGITAL SOCIETY AND DIGITAL BUSINESS WORK; THEY ARE THE BASIS OF EVERY DIGITAL STRATEGY." "The 10 Things CIOs Need to Know About APIs and the API Economy" – Gartner (G00318859, Jan 2017) Image courtesy of https://www.instarem.com/blog/the-api-economy-in-2019-hype-or-the-engine-that-fuels-business-growth/
- 4. "DATA IS THE LIFEBLOOD OF YOUR BUSINESS" 4 "Data is the Lifeblood of Your Business – That's Why You Need a Data Strategy at its Heart" – The Drum (Feb 2020)
- 5. FORBES: MAKING THE MOST OF YOUR DATA Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data 5 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
- 6. FORBES: MAKING THE MOST OF YOUR DATA Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data 6 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
- 7. FORBES: MAKING THE MOST OF YOUR DATA Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data 7 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
- 8. FORBES: MAKING THE MOST OF YOUR DATA 8 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021) Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data
- 9. FORBES: MAKING THE MOST OF YOUR DATA 9 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021) Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data
- 10. FORBES: MAKING THE MOST OF YOUR DATA 10 "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021) Leave your data where it is. Make sure you can handle all data types. Harness data to drive automation. Democratize data. Protect your data
- 11. 11 INTEGRATION THE PLUMBING YOU CAN'T LIVE WITHOUT
- 12. 12 "How to Justify Strategic Investments in Integration Technology" – Gartner (G00385596, May 2019 )
- 13. 13 Data Integration APIs BRINGING IT ALL TOGETHER: Integration provides the link, but… • APIs are still coupled to the data sources • Availability of backend systems is a dependency • Complexity & cost of APIs designed to aggregate data sources
- 14. 14 Data Integration APIs BRINGING IT ALL TOGETHER: DIGITAL INTEGRATION HUB "An advanced application architecture that aggregates multiple back- end system of record data sources into a low-latency and scale-out, high- performance data store." "Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018) DIH
- 15. WHAT IS A DIGITAL INTEGRATION HUB? 15
- 16. 16
- 17. 17
- 18. • Consolidates and aggregates data from multiple sources • Transforms the data into non-proprietary semantics, using entity views • Allows for the access and manipulation of data without impacting the core business systems • Provides advanced search capabilities • Potential components: • Data warehouse • Data lake • Master Data Management (MDM) • Analytical systems & AI The heart of the DIH HIGH PERFORMANCE DATA STORE 18
- 19. • Captures data at the source (preferably as it happens) • Accommodates multiple integration patterns & styles • Event brokering / messaging • Extract Transform & Load (ETL) • Change Data Capture (CDC) • Integration Patterns (ESB, iPaaS) • Stream processing (Spark, Flint) Gathering your data EVENT-BASED INTEGRATION LAYER 19
- 20. • Microservices access the data at the entity level • Can be hierarchical and inter-connected • Generally do not access the Systems of Record directly • API Gateway provides management and access layer: • Discovery • Security • Abstraction • Monetization • Analytics Surfacing/exposing your data FRONT-END API SERVICES 20
- 21. BENEFITS • Responsive user experience • "Defending" systems of record • 24/7 support • Decoupling the front-end layer from the system of record applications • Supporting legacy systems replacement • Normalising the APIs for a certain application domain • Providing real-time business insight
- 22. CHALLENGES • Complexity of rolling out a high-performance data management technology (e.g. NoSQL DBMS or in-memory data grid) • Supporting bidirectional, event-driven synchronization between the high-performance data store and system or record applications • Designing a canonical data model for the DIH business entities that supports multiple channels • Implementing appropriate metadata management to support discovery and introspection of data entities and relationships represented across multiple data sources • Designing, building, and managing the complex distributed architecture of a Digital Integration Hub 22
- 23. RECOMMENDATIONS 23 Determine if your organisation needs a DIH and if it has the skills to support it Know your data and consumer requirements Understand the integration patterns that will be required Design your APIs to be abstracted from the underlying systems ("API First" approach) Consider using a technology partner for implementation
- 25. AZURE DIH ACCELERATOR • Template on GitHub • Jump starts by setting up the engineering system for your application • Allows you to concentrate on the business logic • Consists of: • A pre-configured development environment • An application • Build and Deployment automation 25 https://github.com/Azure-Samples/digital-integration-hub
- 28. GOALS: • Establish an integration platform • Integrate core clinical systems using HL7 while enabling integration of modern digital consumers over FHIR APIs • Enable the interoperability of internal and external client systems hosted in a fully controlled on-premises infrastructure or in a secure public cloud Pathology Services Provider CASE STUDY 28 https://customers.microsoft.com/en-us/story/1384019799287777382- nsw-government-health-provider-azure
- 30. SUMMARY • API-based access to disparate data services is costly from a performance and maintenance perspective • A DIH enables enhanced performance in accessing your organisational data, while ensuring effective protection of your backend systems • A DIH architecture also provides increased scalability, greater flexibility, and better insights
- 31. REFERENCES "Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018) "The 10 Things CIOs Need to Know About APIs and the API Economy" – Gartner (G00318859, Jan 2017) "Data is the Lifeblood of Your Business – That's Why You Need a Data Strategy at its Heart" – The Drum (Feb 2020) "Data Hubs, Data Lakes & Data Warehouses: How They Are Different and Why They Are Better Together" – Gartner (G00465401, Feb 2020) "Five Ways to Make the Most of Your Data" – Forbes (Jul 2021) "How to Justify Strategic Investments in Integration Technology" – Gartner (G00385596, May 2019 ) "Infuse your data hub strategy with Data & Application Integration" – Gartner (G00343327, Dec 2017) 31
Notas del editor
- APIs are can solve many problems: Enable systems & applications to interact Disparate data sources & access protocols Disparate consumer channels But… there be challenges: Security Performance (consumer & system impact) Avoiding numerous point-to-point solutions
- APIs only as good as the data they can access Large organisations rely on multiple systems of record Information needs to be shared on multiple channels Other internal systems Partner systems Client systems Strategy is important
- Connect rather than collect – for real-time insights
- structured, unstructured, relational, and geospatial, etc. Make sure you can analyze and knit together these different types into a unified whole. modern platform treats data as multimodal and multidimensional.
- use it to drive automation and machine-augmented decision making. Invoice reconciliation, handling basic customer complaints
- empower people with self-service analytics tools
- take into account any areas of exposure or vulnerability and build in the right security, access controls and governance to protect your data
- What does Gartner have to say about integration?
- Let's break down the components
- High cost: huge number of API calls potentially imply a massive, often low-value workload which ultimately hits the system of record applications and the integration layer. ■ Complexity: The integration macroservices could be extremely complex to develop, deploy, run and manage, especially if the back-end data is highly fragmented across multiple applications and data sources. Availability: Dependency on backend systems being available Tight coupling: Changes in the back-end systems imply re-engineering of the integration macroservices or their complete redesign, for example, if a back-end system is replaced with a different solution.
- "By storing an aggregated replica of the system of record data needed by the channel applications, the DIH protects the former from excessive workloads while optimizing the data access latency and responsiveness for the latter."
- Metadata management – enables metadata-driven development & introspection, e.g to support domain-driven design Discover, capture, & synchronise metadata models Analytics – technically not part of the DIH, but often a driver for its implementation
- ETL – preferred for initial loads or periodic refreshes
- Some exceptions to direct access: CQRS (Command Query Responsibility Segregation) pattern Updates to SoR Would typically use data virtualization, federated views
- Responsive user experience: while enabling users access to a consolidated, yet real-time, view of data scattered across multiple system of record back-ends "Defending" systems of record: from the potentially excessive, often low value, workloads generated by the channels. 24/7 support: Always-on access to the front-end API services, even in situations where the back-end systems must be put offline for maintenance, upgrade or other reasons. ■ Decoupling the front-end layer from the system or record applications: Supporting legacy system of record applications modernization: Normalizing the APIs for a certain application domain: So that with a single set of APIs the channel applications can access data held in multiple systems of record Providing real-time business insight: Gathering data on user behavior or offering them additional services (including search, predictive analytics
- DRIVER: low-latency, real-time, consolidated and accurate access to data is critical for business success.
- This repository contains a template of an application built for the Azure Application Platform. This template is built to make it easy to dive straight into implementing business logic without having to spend time on setting up an engineering system for your application. The templates give you a starting point, while providing the option to change and extend any of the pre-configured components to suit your needs.
- The front end processing layer are implemented by Azure API Management for providing the discoverability and gateways for all your APIs. The microservices implementation leverage serverless functions which provide a simple CRUD based API. The data layer is implemented using PostgresDB For Analytics and getting a single pane of glass view over your disparate data sources this is where you could optionally add on Azure Synapse analytics to extend this solution. This is not included in the sample. The Integration layer in this case uses Azure Event Grid for the event driven programming model to ensure the system responds to all API events in real time and Logic Apps to react upon these events and process the incoming data. The implementation of the logic app is left blank, leaving you to leverage the 450+ connector ecosystem to implement the syncing functionality between the data layer and your backend systems of record you choose to integration the Digital Integration Hub with. and Monitoring provided by Application Insights and Log Analytics features of Azure Monitor. The application is comprised of an Azure Function API implemented written in Typescript, a PostgreSQL 'items' database and an Integration layer implemented by Logic Apps and Event Grid. It uses an Object-Relational Mapper (Sequelize) and implements a single object, items, to get you started.