This was a very interesting conference, TIC students oriented where I take him to the azure ecosystem for data warehousing architecture and best practices to reach powerful Business Intelligence Solutions according to the new era

1. Building a Modern
Data Warehouse
Elena López
Microsoft MVP – Data Platform

2. ¿Quién es Elena López?
Ing. de Sistemas
Datos, datos y más datos
Curiosa, emprendedora, autodidacta
Humor, cerveza, naturaleza

3. Aprendiendo BD con SQL Server Elena López + SQL Server

4. ¿Qué es un
Data Warehouse?

5. ¿Desaparecerá el
Data Warehouse ?
Es necesario ahora más que nunca:
 Integra múltiples fuentes de datos
 Disminuye el impacto negativo de
reportes a producción
 Análisis histórico de los datos
 Estructura amigable
 Erradica los silos
 Brindan una única versión de la verdad

6. RETOS
(Nuestros)
Nosotros Los Datos

7. ¿Qué hace moderno a
un Data Warehouse ?
 Procesamiento de grandes volúmenes
de datos
 Capacidad de procesar datos casi en
tiempo real y a gran velocidad
 Apoya el auto-servicio
 Fomenta la democratización de la data
 Facilita la exploración de los datos
 Visualización dinámica
 Infraestructura híbrida o en la nube

8. Modern Data Warehouse
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
Azure Data Factory
SSIS
Azure Data Lake
Storage Gen2
Blob Storage
Cosmos DB
Azure Databricks
Azure HDInsight
Power BI Dataflow
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
Cosmos DB
Power BI Aggregations

9. Ingest – Data Orchestration and Monitoring

10. Modern Data Warehouse
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
Azure Data Factory
SSIS
Azure Data Lake
Storage Gen2
Blob Storage
Cosmos DB
Azure Databricks
Azure HDInsight
Power BI Dataflow
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
Cosmos DB
Power BI Aggregations

11. Store – Big Data Store

12. Modern Data Warehouse
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
Azure Data Factory
SSIS
Azure Data Lake
Storage Gen2
Blob Storage
Cosmos DB
Azure Databricks
Azure HDInsight
Power BI Dataflow
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
Cosmos DB
Power BI Aggregations

13. A “no-compromises” Data Lake: secure, performant, massively-scalable Data Lake storage that brings the cost and
scale profile of object storage together with the performance and analytics feature set of data lake storage
Azure Data Lake Storage Gen2
M A N A G E A B L E S C A L A B L EF A S TS E C U R E
 No limits on
data store size
 Global footprint
(50 regions)
 Optimized for Spark
and Hadoop
Analytic Engines
 Tightly integrated
with Azure end to
end analytics
solutions
 Automated
Lifecycle Policy
Management
 Object Level
tiering
 Support for fine-
grained ACLs,
protecting data at the
file and folder level
 Multi-layered
protection via at-rest
Storage Service
encryption and Azure
Active Directory
integration
C O S T
E F F E C T I V E
I N T E G R AT I O N
R E A D Y
 Atomic file
operations
means jobs
complete faster
 Object store
pricing levels
 File system
operations
minimize
transactions
required for job
completion

14. Objectives
 Plan the structure based on optimal data retrieval
 Avoid a chaotic, unorganized data swamp
Data Retention Policy
Temporary data
Permanent data
Applicable period (ex: project lifetime)
etc…
Business Impact / Criticality
High (HBI)
Medium (MBI)
Low (LBI)
etc…
Confidential Classification
Public information
Internal use only
Supplier/partner confidential
Personally identifiable information (PII)
Sensitive – financial
Sensitive – intellectual property
etc…
Probability of Data Access
Recent/current data
Historical data
etc…
Owner / Steward / SME
Subject Area
Security Boundaries
Department
Business unit
etc…
Time Partitioning
Year/Month/Day/Hour/Minute
Downstream App/Purpose
Common ways to organize the data:
Organizing a Data Lake – Folder structure

15. Prep – Transform and Clean

16. Modern Data Warehouse
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
Azure Data Factory
SSIS
Azure Data Lake
Storage Gen2
Blob Storage
Cosmos DB
Azure Databricks
Azure HDInsight
Power BI Dataflow
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
Cosmos DB
Power BI Aggregations

17. Model & Serve – Data Warehouse

18. Modern Data Warehouse
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
Azure Data Factory
SSIS
Azure Data Lake
Storage Gen2
Blob Storage
Cosmos DB
Azure Databricks
Azure HDInsight
Power BI Dataflow
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
Cosmos DB
Power BI Aggregations