IdealNet Data Integration ETL vs Cloud

2933 West Germantown Pike
Building 2, Suite 204
Fairview Village, Pa 19409
USA
Toll Free : 800 618-0836
Fax : 610 666-1006
Email : info@idealnetinc.com
Toll Free : 800 618-0836
Fax : 610 666-1006
June 2011
Data Integration Study and Results:
ETL Versus Cloud Based Data Integration
Christopher C. Biddle

© 2011 IdealNet, Inc.Page 2
Taking Your Business to the Next Level™
IdealNet, Inc.Page 2
Agenda
• IdealNet Corporate Overview
• Data Integration Study – Background & Drivers
• State of Data Integration Market
• Customer Challenges
• Comparison Categories
– Business Application
– Platform Deployment
– Connectivity to Data Sources
– Synchronization
– Transformation
– Data Movement
– Test, Development and Operations Environments
– Data Modeling
– Data Quality & Data Governance
– Architecture and Standards
• Conclusions
Agenda

IdealNet Corporate Overview
• Ideal Systems, Inc. was formed in 1994 : provided Enterprise Software and
consulting
• IdealNet, Inc. was formed in 2002 and absorbed the Ideal Systems, Inc.
consulting practice
• Provides expert Business and Technical consulting services to Life Sciences
and Financial Institution customers
• Clients include all of the top 10 Tier 1 Pharmaceutical Manufacturers, Leading
Financial Institutions including Investment Banks and Commercial
Banks, Biotech, Medical Device, Hospital Group Purchasing Organizations
(GPOs), and Distributors
• Areas of expertise include Commercial and Regulatory
Contracting, Finance, Merger & Acquisition Support, Trading
Strategies, Business Intelligence & Analytics, and Sales Force Automation
• Application Development and Customization, Master Data Management and
Application Integration
• Many years of experience in implementing, integrating, and upgrading Enterprise
Software solutions
• Based on the East Cost in a Philadelphia, PA suburb with clients throughout the
world

IdealNet Partial Client List

Background & Drivers
• User Issues
– Applications are growing & complexity is
increasing
– Bridging these islands is becoming
increasingly more important
– Number of “Bridges” required is growing in
leaps and bounds – it’s a mess
– Need for unifying key data across applications
such as customer, product, human relations
and vendor data
– On-premise and Cloud are very different
– Cloud is coming to large enterprise (with the
right security) as Salesforce® leads the
– Compliance is always an issue with any public
company – ETL makes it hard
– Strong ROI associated with the results – so
users go forward

1st Generation
ETL
Relational
Manual Discovery
“A” to “B” Pipe
4:1
2nd Generation
ETL with GUI
Wire Diagrams
Relational
Manual Discovery
Simple Connectors
“A” to “B” Pipe(s)
MDM Hub
4:1
Some Cloud Apps
3rd Generation
Persistent Metadata
Server – Hub & Spoke
Data Virtualization
Technology
Automated Discovery
Automated Operations
Relational, Object, XML,
NoSQL
No Programming
Virtual MDM™
2:1 to perhaps 1:1
Full Cloud & On-Premise
Data Integration Technology Curve
1985
Dominant
Technology
in Use Today
2000 2010
Technology Advances

Terminology
• Terminology Varies Considerably
– ETL (extract/transform/load) – this is the most consistent term
– Data Integration, EAI (transaction level), EII (failed in the market - extinct), ESB
– Data Virtualization (instance), Advanced Data Virtualization (persistent
metadata server)
– Data Alignment, Data Synchronization, Data Harmonization
– Master Data Management, Virtual Master Data Management
– Cloud: Iaas (integration as a service), Iaas (information as a service), iPaas
(information platform as a service)

Market Size - Billions of Dollars
• IDC® - Data integration and Access Software 2009
– Published December 2010
– 2010 Projected - $3,450 billion
• Overall 7.8% cagr thru 2014 reaching total market size of $4,759 billion
• Gartner ®– Magic Quadrant for Data Integration Tools
– Published November 2010
– 2010 Projected - $1,431 billion (up 6%)
• Overall 9.4% cagr thru 2014 reaching total market size of $2,000 billion
• Forrester ®– Information As A Service - Q1 1020
– Published February 2010
– “The current IaaS market size is $3,300 billion”
• “and is likely to grow to more than $6,700 billion by 2012, a 40% annual growth rate”
• MDM market tracked separately by most
– $1- 2 billion perhaps – this market suffered a major setback over last 24 months
– Virtual MDM™ is coming
• Gartner® DataQuest – April 30 Report on IT Services
– $73 billion in IT services dollars spent worldwide in 2009 … how much is data
integration?

Market Metrics and Trends
• Gartner®
– “35% of all large and midsize organizations worldwide will be using one or
more iPaaS (cloud based integration Platform as a Service ) offerings in some
form by 2016.” – 2011
• Enterprise Strategy Group®
– “Data integration platforms must be able to integrate various types of data in
the cloud, along with data on-premises and in remote locations.” – June 2011
• Forrester®
– “Integration by ETL creates data quality problems and delays information
delivery.” – June 15, 2011

Business Application
• Key Parameters
– Data Integration – regular movement of data
– Data Migration – one time movement of data
– Business Intelligence - loading the BI vendor “domain”
– Data Warehouse, Data Mart
– Master Data Management – complex and difficult movement of data
– Virtual Master Data Management – new version of MDM based upon advanced data
virtualization technology
• Cautions
– One “shoe does not fit all”
– Cloud, Object Applications in the Cloud don’t fit well with ETL based technologies
• Best Practice
– Get a metadata discovery tool (note the free Queplix offering amongst others)
– Define clearly the data sources, amount of data to be moved, the keys that will help
make this work, the frequency of update, and the exact minimum of information to be
moved/synchronized/federated prior to selecting a technology

Platform Deployment
• Key Parameters
– On-premise
– Cloud (Public and/or Private)
– Software as a service (saas)
• Cautions
– ETL tools are not designed for cloud models or software as a service
– This includes large enterprise private cloud - problems are not limited to public
cloud
– No basic security mechanisms in ETL for cloud deployment
– Technologies such as QueCloud support VPN integration and optionally a
special security module that implements enterprise model for cloud to
enterprise connectivity
• Best Practice
– Understand the full potential for architectural expansion up front
– ETL will not adapt to expanded horizons and this will create large problems for
you later

Connectivity to Data Sources
• Key Parameters
– Direct database connectivity (insert/update/delete)
– Application program interface (access only through API)
– Flat file (EDI), Standards
• Cautions
– Programming to proprietary vendor API’s on legacy software such as SAP®, Siebel®,
PeopleSoft® requires extensive application specific skills and may substantially increase
the cost of a project
– Stick with SOA
– Custom fields – how will you know they exist? How will your vendor handle this?
– Cloud based applications are object oriented – not relational – find a toolset that works
well with objects – not just relational tables
• Best Practices
– Use a metadata discovery tool (2nd time)
– Use intelligent interfaces that eliminate the need to know a vendor API (Application
Software Blades™ - Queplix)
– No vendor has every interface or ever will – understand their strategy to add your
application and ask them to do it!

Synchronization
• Key Parameters
– Batch
– Real-time or Near Real-time
– Other
• Cautions
– Real-time may mean an ACID transaction – you are in a transaction flow and
this is a very different problem than customer data alignment – this is really
enterprise application integration as this sort of integration requires application
level changes, not just data integration at the database level
– ETL often requires you set triggers in a database – database administrators
don’t like this
• Best Practice
– Update (synchronize or harmonize) when a “record of truth” changes and then
only update the fields that need updating – you don’t need to update an entire
“row” if you choose the right technology set
– Understand strategies for real-time and near-time that don’t require explicit
database invasive triggers

Transformation
• Key Parameters
– String, Math, Boolean, Fuzzy Logic
– Complex, Summarization, Statistical
– Custom Transforms
– 1st Generation Programming and SQL
– 2nd Generation GUI Programming and SQL
– 3rd Generation Automation, Check Box and Configuration, Excel®-like formula
builder
• Cautions
– 1st and 2nd Generation products don’t mix well with business users – expect to
be programming and working with lots of details
• Best Practice
– Find a 3rd Generation product – this should work like Excel® - transforms are
just simple formula’s

Data Movement
• Key Parameters
– Bulk Data Movement
– Data federation for BI deployment or equivalent
– Record level synchronization
• Cautions
– Map the problem to the technology solutions – once again fit is key
• Best Practice
– ETL is the workhorse file mover for batch windows
– Record level manipulation for data integration is best achieved by Advanced
Data Virtualization based products

Test, Development and Operations
• Key Parameters
– Target User, Dashboard, Workflow
– Security, Data Quality
– Data dictionary – metadata repository
– Shared library of transforms
– Test, Development and Production environment
– Reporting, Analytics, Graphics
– Disaster recovery
• Cautions
– How will your vendor support flow from test & development into production operations?
Automated back-up?
– Integration with a software instance is significantly more limiting than a server based
product with persistence
– Hub and spoke architectures require LDAP support and more – ETL doesn’t do this
• Best Practice
– Understand the difference between test, development and production operations
environments - SMB customers need to understand this better

Data Modeling
• Key Parameters
– Connection to data sources
– Discovery of metadata structure in data sources
– Representation of metadata structures in data sources
– Automatic update of metadata in data sources
– Semantic discovery support
– Search of metadata across multiple sources
– Direct access to underlying data – in a useful and navigable format – from
metadata
– Ability to model all data sources
– Virtual structures in metadata to facilitate mapping
– Lineage of metadata and Metadata export
• Cautions
– This should all be integrated – metadata discovery, data dictionary, semantic
discovery, data catalog access, data integration transforms
• Best Practice
– Get a metadata discovery tool (3rd time!)

Data Quality and Data Governance
• Key Parameters
– Basic data quality deeply integrated with data integration process
• Development and source clean-up
• Production environment – ongoing automation?
• The Cloud ***MUST*** support data quality – or you don’t have a solution
– Data governance – implementation by business rule
• Cautions
– Data quality problems derail business intelligence and data integration all the
time – don’t let it happen to you
– Some 1st and 2nd Generation and most 3rd Generation products integrate data
quality – separate products, except for the largest organizations, probably don’t
make sense
– Gartner® has noted the convergence of data quality with data integration and
other toolsets
• Best Practics
– Data integration proposals, without completely addressing data cleansing, data
quality and associated data governance guidelines won’t produce the results
you expect (timeframe, quality, return on investment)

Architecture and Standards
• Key Parameters
– Standalone
– Networked
– Scale from simple integration to virtual master data
management
– SOA, JDBC, ODBC-.Net, Web Services
– Other
• Cautions
– ETL links are standalone “A to B” instances of connectivity –
as these multiply how will you vendor “manage” all of this?
– Standards are standards – if SOA cost more that’s a big red
flag
• Best Practice
– Use standards
– Embrace enterprise strategies for data integration
“management” if it applies to you

Conclusions
• Established ETL Vendors
– Informatica® is the leader of the pack for ETL – see release 9.1
– Clearly wins and defines ETL for batch oriented bulk file transfer
– What does Hadoop® file level integration support actually deliver? Missing Hadoop®
resident database support so there is not much today.
– Talend® and Pentaho® are the open source leaders with associated ETL and BI
strategies – these are excellent choices.
– ETL doesn’t really scale – hubs for MDM are all manually programmed, expensive to
setup and error prone – each connection is essentially standalone
• Data Virtualization Based Data Integration
– Queplix technology is a solid 3rd Generation product which merits your review – they are
clearly the leader in data virtualization today
– QueCloud (or the on-premise product, Virtual Data Manager™) integrate 2, 3, 4 or more
sources as uniquely enabled by advanced data virtualization – this reduces risk, lowers
cost and provides much more capability
– Lower costs by 50% or more and increase savings as you add more application
integrations
– Data virtualization is the future of data integration – persistent metadata servers do
things that ETL technology cannot do – learn more about it

Sponsored Message From Queplix
• Sponsored Message (Queplix, Inc.)
– Please go to www.queplix.com and download the Free Metadata Discovery
Tool or email sales@queplix.com and they will set you up for free
– Queplix will email out a copy of my report or follow this link to obtain a copy
directly from my website:
http://www.idealnetinc.com/IdealNet_Analysis_of_Data_Integration_Technologi
es.pdf
– Queplix is running promotions for free QueCloud implementation turnkey –
essentially all setup and 1 year free – please contact sales@queplix.com for
the terms and conditions or see www.netsuite.com for more about that
promotion in the partner section
– Queplix has free video, without registration, accessible from their home page
and from their collateral page (under “About”)
– Queplix has other video and white papers in a registration section
• This Presentation
– You can request a copy via email from me cbiddle@idealnetinc.com and my
team will follow-up promptly

2933 West Germantown Pike
Building 2, Suite 204
Fairview Village, Pa 19409
USA
Toll Free : 800 618-0836
Fax : 610 666-1006
Toll Free : 800 618-0836
Fax : 610 666-1006
Contact Christopher C. Biddle at cbiddle@idealnetinc.com
Thank You
Visit us online at www.idealnetinc.com

Business Application & Platforms

IdealNet Data Integration ETL vs Cloud

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (7)

Similar a IdealNet Data Integration ETL vs Cloud

Similar a IdealNet Data Integration ETL vs Cloud (20)

Último

Último (20)

IdealNet Data Integration ETL vs Cloud

Notas del editor