Watch full webinar here: https://bit.ly/35sp2Q0
Success or failure in the digital age will be determined by how effectively organisations manage their data. The speed, diversity and volume of data present today can overwhelm older data architectures, leaving business leaders lacking the insight and operational agility needed to respond to market opportunity or competitive challenges.
With the pace of today’s business, modernisation of a data architecture must be seamless, and ideally, built on existing capabilities. This webinar explores how data virtualization can help provide a seamless evolution to the capabilities of an existing data architecture without business disruption.
You will discover:
- How to modernise your data architectures without disturbing the existing analytical workload
- How to extend your data architecture to more quickly exploit existing, and new sources of data
- How to enable your data architecture to present more low latency data
3. ▪ RXP Data Management Framework
End-to-end Information Management framework for the entire data lifecycle.
Data Virtualisation is
relevant in these areas
7. DATA CONSUMERS
DISPARATE DATA SOURCES
SQL Queries
(JDBC, ODBC, ADO.NET)
Web Services
(SOAP, REST, OData)
Web-based catalog
& search
Secure delivery
(SSL/TLS)
DATA CONSUMERS
MPP Processing
Relational Cache
Corporate Security
Monitoring & Auditing
Metadata
Repository
Execution Engine
& Optimizer
Data Virtualization as a Data Access Layer
DATA VIRTUALIZATION
Consume
Combine
2
3
Connect
1
8. DATA CONSUMERS
DISPARATE DATA SOURCES
SQL Queries
(JDBC, ODBC, ADO.NET)
Web Services
(SOAP, REST, OData)
Web-based catalog
& search
Secure delivery
(SSL/TLS)
DATA CONSUMERS
Data Virtualization in Action
Consume
Combine
2
3
Connect
1
Base/Raw views
Standardized
views
Customer Product Order
Business viewsFinance Operations Sales
Less Structured
Operational
Each Layer of Views
provides more refined
Single Views of Truth
10. 10
Demo Scenario
▪ Historical sales data offloaded to Hadoop
cluster for cheaper storage
▪ Marketing campaigns managed in an external
cloud app
▪ Country is part of the customer details table,
stored in the DW
Sources
Combine,
Transform
&
Integrate
Consume
Base View
Source
Abstraction
join
group by state
join
Sales Campaign Customer
SaaS solution
How effective are our marketing Campaigns?
15. 15
What is the optimizer doing?
SELECT c.state, AVG(s.amount)
FROM customer c JOIN sales s
ON c.id = s.customer_id
GROUP BY c.state
Sales Customer
join
group by
Sales Customer
Create temp
table
join
group by
Option 1?
Option 2? Option 3?
Temp_Customer
Customer and Sales are in different sources.
What is the best execution plan?
Naïve Strategy Temporary Data Movement
300 M 2 M
2 M
50 M
Sales Customer
join
group by ID
Group by
state
Partial Aggregation Pushdown
2 M
2 M
‘Cost’ ~302 M ‘Cost’ ~52 M ‘Cost’ ~4 M
16. 16
Why is this so important?
SELECT c.name, AVG(s.amount)
FROM customer c JOIN sales s
ON c.id = s.customer_id
GROUP BY c.state
How Denodo works compared with other federation engines
System Execution Time Data Transferred Optimization Technique
Denodo 9 sec. 4 M Aggregation push-down
Others 125 sec. 302 M None: full scan
300 M 2 M
Sales Customer
join
group by
2 M
2 M
Sales Customer
join
group by ID
Group by
state
To maximize push
down to the EDW
the aggregation is
split in 2 steps:
• 1st by customerID
• 2nd by state
This significantly
reduces network
Traffic and processing
In Denodo
17. 17
Denodo Performance Strategies
• Post-processing and Federation in the DV engine
• Delegation
▪ Process as much as possible in the data sources
• Temporary Tables
▪ Automatically move data to the biggest data source to optimize the execution
• Summaries
▪ Based on the query the Denodo optimizer can use a “summary” for accelerating the execution
• MPP Integration
▪ Move processing to an external MPP system on the fly
• Caching
▪ Persist data beforehand in a relational database
18. Next Steps
Join us again for the final topic in this 3-part series :
Webinar 3 will be held 12:30 Thursday 19th November
▪ Topic: Data Virtualisation for Business Consumption
Contact information:
Adrian Bridge
Principal Consultant
RXP Group
0417 875 919
adrian.bridge@rxpservices.com
Katrina Briedis
Sales Engineering
Denodo
+61 450 499 440
kbriedis@denodo.com