This document discusses modernizing a company's data architecture using data virtualization to improve agile data delivery. It describes the company's current federated model and need to respond more quickly. The proposal is to implement a data virtualization layer to land data from various sources, create a conformed data warehouse, and provide a single data access point. This would virtualize data, reduce costs, improve flexibility and throughput, and accelerate change cycles compared to the current environment. A specific example of using this approach to enable daily and intra-day reporting from a new system is also described.
Exploring the Future Potential of AI-Enabled Smartphone Processors
Modernizing Data Architecture using Data Virtualization for Agile Data Delivery
1. INTERNAL USE ONLY
Modernizing Data Architecture using Data Virtualization for Agile
Data Delivery
Dave Kay, Data Consultant
UKGI Data, Analytics and Architecture
3. INTERNAL USE ONLY 3
55,000Employees (approximately)
170+
Countries
USD 4.6 bn
Business operating profit
4. INTERNAL USE ONLY 4
The Zurich Group Operates a Federated Model for Data
Deliveries.
Group Function – Delivers Data Framework
Satellite Function – ZNA
Satellite Function – UKGI & UK Life
Satellite Function – Zurich Switzerland BU
Etc…
10. INTERNAL USE ONLY 10
SchedulingDataSources
ETLProcessing
SolutionDelivery
UserAccess
11. INTERNAL USE ONLY 11
Reduce
Cost
Flex
Resource
Raise
throughput
Accelerate
change
• Cost of change prohibitive to generating value from data
• Slower time to market inhibiting impact of good analytics
• Complexity of solution can cause delivery bottlenecks
• Bespoke coded solution requires unique resource to maintain
12. INTERNAL USE ONLY 12
Data
Landing
Conformed
Warehouse
Data Hub Interface
Layer
• Data landed into infrastructure from disparate sources
• Conformed enterprise data warehouse provides central storage
• Data hub providing single ‘user facing’ data source
• Simplified Interface layer landscape – single point of entry
14. INTERNAL USE ONLY 14
Business Need: Provide daily MI and intra-day
Operational Reporting for a community of business
analysts, salesmen, underwriters, managers and
actuarial staff – off the back of a brand new system.
• XML Source with over 13,000 elements, 10+ products
• XML Schema not available
• Data items and schema differ dramatically per
product type
• Fast pace of change
15. INTERNAL USE ONLY 15
• Subset of data ‘shredded’ via XSLT – immediately reduces scope
(250 items)
• SSIS transforms at multiple points generate maintenance difficulties
• Data effectively replicated multiple times
• Change Request process lengthy and expensive
• Specialist development skills required
16. INTERNAL USE ONLY 16
• XML made available into new DAA architecture for ingestion
• Denodo used as a virtual staging wrapper
• Views over XML representing core entities for reporting (Relational
View) virtualised in Denodo – cached to SQL Server
• Relational, Reporting and Analytical stores published as
consumable data sets via Denodo
• Two tier interface:
• Rapid interactive access to quickly changing data for ad-hoc
queries via Alteryx, JDBC, REST etc
• Stable, conformed relational marts virtualised & cached for
exploration, analysis and visualisation in Qlik etc
19. INTERNAL USE ONLY 19
DataSources
VirtualStaging
VirtualSemantic
AtomicWarehouse
VisualisationAnalytics
20. INTERNAL USE ONLY 20
Reduce
Cost
Flex
Resource
Raise
throughput
Accelerate
change
• Cost model reduced
• Significantly quicker time to market
• Greater throughput
• Resource pool expanded