SlideShare a Scribd company logo
1 of 40
Leading Practices and Insights for Managing Data Integration Initiatives May 7, 2010
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
About Allin ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
About Optima ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Data Integration Overview 03/09/11 Definition  - combining data residing in different sources and providing users with a unified view of these data Mediated Schema Example Data Warehouse Example
Business / Technical Drivers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Business / Technical Drivers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Key Variables / Considerations 03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Approaches / Strategies 03/09/11
03/09/11 ,[object Object],Approaches / Strategies
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],Approaches
Application Specific Solutions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 Data Propagation ,[object Object],[object Object],[object Object],[object Object]
Data Federation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Integration Tools 03/09/11
03/09/11 ETL ETL: Extract, Transform and Load ,[object Object],[object Object],[object Object],[object Object]
ETL Integrated Architecture XYZ Corp Systems  Data Extraction &  Integration Business Process Layer Information Management Presentation Layer Accessible throughout the organization Distribution Ad-hoc Analysis Tools Core Business Systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Architecture ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Reporting Data   Storage Transformation Subject Areas Transformation Extract, Transformation, & Load (ETL) Layer The “Unified Business Model” and Information Management Analytics and Reporting Tools OLAP Cubes &  Predictive Models ,[object Object],[object Object],[object Object],[object Object],Exception Notifications Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 EAI ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 EAI Architecture
03/09/11 Tool Comparison ,[object Object]
03/09/11 ETL Tools - Key Features ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Extraction &  Integration Data   Storage Transformation Transformation Extract, Transformation, & Load (ETL) Layer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 ETL Tools – Key Features  (continued) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Extraction &  Integration Data   Storage Transformation Transformation Extract, Transformation, & Load (ETL) Layer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Challenges 03/09/11
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Challenges
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Challenges  (continued)
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Integration Challenges  (continued)
Case Studies 03/09/11
Business Case 1 03/09/11 Service Company – Revenue: $200m, Size: 300 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 03/09/11 Business Case 2 Education Company – Revenue: $500m, Size: 300 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 Business Case 3 Healthcare Company – Revenue: $150m, Size: 200 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions 03/09/11
The Value of a Data Quality Effort  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
The Value of a Data Quality Effort  ,[object Object],[object Object],[object Object],[object Object],03/09/11
Importance of Governance ,[object Object],Size 03/09/11 Incorrect hardware or software 7% 2% Failure to define objectives 17% Unfamiliarity with scope and complexity 17% Lack of communication 20% Inadequate Project Management 32% Other 5%
Data Stewardship Data stewards act as the conduit between IT and the business and accept accountability for the data management process.  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Stewards play a the central role in the management of data across the organization and in assuring its usefulness for the business. 03/09/11 Data stewards become the “public face” for data and have the following responsibilities: IT Business Business Data  Stewards
Success Factors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Success Factors  (continued) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Appendices 03/09/11
03/09/11 ETL Vendors ETL Vendors ETL Tools Microsoft   SQL Server Integration Services  Oracle   Oracle Warehouse Builder (OWB) SAP Business Objects Data Integrator & Data Services  IBM IBM Information Server (Datastage) IBM Data Manager/Decision Stream (Cognos) SAS Institute  SAS Data Integration Studio Informatica   PowerCenter Ab Initio Co>Operating System Information Builders Data Migrator Adeptia Adeptia Integration Server CastIron Systems OmniConnect Platform Pitney Bowes Business Insight DataFlow Manager Pervasive Data Integrator Elixir  Elixir Repertoire Javlin   Clover ETL Pentaho   Pentaho Data Integration  Talend  Talend Open Studio
03/09/11 ETL / EAI - Tool Strengths ETL  EAI  Excels at bulk data movement  Limited in data movement capabilities  Provides for complex transformations, aggregation from multiple sources and sophisticated business rules.  Offer less sophisticated transformation and extraction functions  Assumes data delays.  Operates in real time  Are batch-oriented, making them fast and simple for one-time projects and testing  Work better with continuously interacting systems  Offers little in the way of workflow  Workflow-oriented at the core  Works primarily at the session layer  Works primarily at the transport layer

More Related Content

What's hot

What's hot (20)

Data Strategy
Data StrategyData Strategy
Data Strategy
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
 
Straight Talk to Demystify Data Lineage
Straight Talk to Demystify Data LineageStraight Talk to Demystify Data Lineage
Straight Talk to Demystify Data Lineage
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data Factory
 
Azure Data Engineering.pptx
Azure Data Engineering.pptxAzure Data Engineering.pptx
Azure Data Engineering.pptx
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
 
Big Data Architecture
Big Data ArchitectureBig Data Architecture
Big Data Architecture
 
Data Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and GovernanceData Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and Governance
 
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
 
Session découverte de la Data Virtualization
Session découverte de la Data VirtualizationSession découverte de la Data Virtualization
Session découverte de la Data Virtualization
 
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solutionDifferentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
 
Informatica MDM Presentation
Informatica MDM PresentationInformatica MDM Presentation
Informatica MDM Presentation
 
Data Center Migration Roadmap
Data Center Migration RoadmapData Center Migration Roadmap
Data Center Migration Roadmap
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
1- Introduction of Azure data factory.pptx
1- Introduction of Azure data factory.pptx1- Introduction of Azure data factory.pptx
1- Introduction of Azure data factory.pptx
 
Data Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data StrategyData Audit Approach To Developing An Enterprise Data Strategy
Data Audit Approach To Developing An Enterprise Data Strategy
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 

Similar to Managing Data Integration Initiatives

t2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevityt2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevity
Jonathan Hamilton Solórzano
 
Datawarehousing
DatawarehousingDatawarehousing
Datawarehousing
sumit621
 
Rev_3 Components of a Data Warehouse
Rev_3 Components of a Data WarehouseRev_3 Components of a Data Warehouse
Rev_3 Components of a Data Warehouse
Ryan Andhavarapu
 
What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...
kzayra69
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Nathan Bijnens
 

Similar to Managing Data Integration Initiatives (20)

Chapter 2 - Enterprise Application Integration.pdf
Chapter 2 - Enterprise Application Integration.pdfChapter 2 - Enterprise Application Integration.pdf
Chapter 2 - Enterprise Application Integration.pdf
 
DW 101
DW 101DW 101
DW 101
 
t2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevityt2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevity
 
Datawarehousing & DSS
Datawarehousing & DSSDatawarehousing & DSS
Datawarehousing & DSS
 
ETL Market Webcast
ETL Market WebcastETL Market Webcast
ETL Market Webcast
 
Why Data Virtualization? An Introduction by Denodo
Why Data Virtualization? An Introduction by DenodoWhy Data Virtualization? An Introduction by Denodo
Why Data Virtualization? An Introduction by Denodo
 
GROPSIKS.pptx
GROPSIKS.pptxGROPSIKS.pptx
GROPSIKS.pptx
 
Datawarehousing
DatawarehousingDatawarehousing
Datawarehousing
 
A Comparitive Study Of ETL Tools
A Comparitive Study Of ETL ToolsA Comparitive Study Of ETL Tools
A Comparitive Study Of ETL Tools
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....
 
Rev_3 Components of a Data Warehouse
Rev_3 Components of a Data WarehouseRev_3 Components of a Data Warehouse
Rev_3 Components of a Data Warehouse
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Business Analytics System
Business Analytics SystemBusiness Analytics System
Business Analytics System
 
What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...
 
An Integrated ERP with Web Portal
An Integrated ERP with Web Portal An Integrated ERP with Web Portal
An Integrated ERP with Web Portal
 
Data Warehouse Basic Guide
Data Warehouse Basic GuideData Warehouse Basic Guide
Data Warehouse Basic Guide
 
SW Architecture Monolithic to SOA
SW Architecture Monolithic to SOASW Architecture Monolithic to SOA
SW Architecture Monolithic to SOA
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
 
Data warehouse presentaion
Data warehouse presentaionData warehouse presentaion
Data warehouse presentaion
 
Integrating SIS’s with Salesforce: An Accidental Integrator’s Guide
Integrating SIS’s with Salesforce: An Accidental Integrator’s GuideIntegrating SIS’s with Salesforce: An Accidental Integrator’s Guide
Integrating SIS’s with Salesforce: An Accidental Integrator’s Guide
 

Managing Data Integration Initiatives

  • 1. Leading Practices and Insights for Managing Data Integration Initiatives May 7, 2010
  • 2.
  • 3.
  • 4.
  • 5. Data Integration Overview 03/09/11 Definition - combining data residing in different sources and providing users with a unified view of these data Mediated Schema Example Data Warehouse Example
  • 6.
  • 7.
  • 8.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 16.
  • 17.
  • 18.
  • 20.
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 28.
  • 29.
  • 30.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 39. 03/09/11 ETL Vendors ETL Vendors ETL Tools Microsoft  SQL Server Integration Services Oracle  Oracle Warehouse Builder (OWB) SAP Business Objects Data Integrator & Data Services  IBM IBM Information Server (Datastage) IBM Data Manager/Decision Stream (Cognos) SAS Institute SAS Data Integration Studio Informatica  PowerCenter Ab Initio Co>Operating System Information Builders Data Migrator Adeptia Adeptia Integration Server CastIron Systems OmniConnect Platform Pitney Bowes Business Insight DataFlow Manager Pervasive Data Integrator Elixir Elixir Repertoire Javlin  Clover ETL Pentaho  Pentaho Data Integration Talend Talend Open Studio
  • 40. 03/09/11 ETL / EAI - Tool Strengths ETL EAI Excels at bulk data movement Limited in data movement capabilities Provides for complex transformations, aggregation from multiple sources and sophisticated business rules. Offer less sophisticated transformation and extraction functions Assumes data delays. Operates in real time Are batch-oriented, making them fast and simple for one-time projects and testing Work better with continuously interacting systems Offers little in the way of workflow Workflow-oriented at the core Works primarily at the session layer Works primarily at the transport layer

Editor's Notes

  1. 3
  2. 3
  3. Bulk Extract – utilizes copy management tools or unload utilities to extract all or a subset of the operational relational database. The data which has been extracted may be transformed to the format used by the target on the host or target server . The DBMS system load tools are then used in order to refresh the database target. File Compare – process compares the newly extracted operational data to the previous version. After that, a set of incremental change records is created and are applied as updates to the target server within the scheduled process. Change Data Propagation – captures and records the changes to the file as part of the application change process. Techniques that can be used include triggers, log exits, log post processing or DBMS extensions. A file of incremental changes is created to contain the captured changes.
  4. Data stewardship involves taking responsibility for data elements for their end-to-end usage across the enterprise .