How to think about ETL in Cloud Data Lakes using Azure Data Factory and Azure Synapse Analytics

1. Data Lake ETL in the Cloud
with Azure Data Factory
Mark Kromer
Microsoft Azure Principal PM
@kromerbigdata

2. INGEST PREPARE TRANSFORM,
PREDICT
& ENRICH
SERVE
STORE
VISUALIZE
On-premises data
Cloud data
SaaS data
Data Pipeline Orchestration & Monitoring

6. https://techcommunity.microsoft.com/t5/azure-data-
factory/how-to-save-your-data-profiler-summary-stats-in-adf-data-flows/ba-p/1243251

7. • iif (length(title) == 0,toString(null()),title)

13. https://www.youtube.com/watch?v=E_UD3R-VpYE

18. • https://www.youtube.com/channel/UC2S0k7NeLcEm5_IhHUwpN0g
• https://docs.microsoft.com/en-us/azure/data-factory/how-to-data-flow-dedupe-nulls-snippets
• https://www.youtube.com/watch?v=O8gmv5-lXhs
• https://www.youtube.com/watch?v=7gdwExjHBbw
• https://www.youtube.com/watch?v=ryYo8UFUgTI
• https://www.youtube.com/watch?v=E_UD3R-VpYE
• ADF Metafunctions explained
• https://mssqldude.wordpress.com/2020/08/31/adf-data-flow-metadata-functions-explained/