Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Global Azure Norway on April 16th, 2021)

1. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines & Data Flows:
Introduction to Data Integration
in Azure Synapse Analytics
Cathrine Wilhelmsen
Global Azure Norway · April 16th, 2021

2. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Abstract
Do you regularly need to get data for your projects?
Yep! 🙋‍♀️
Data is at the core of every Business Intelligence, Data Science, and Machine Learning project.
You need data to understand what has happened in the past, to predict what may happen in
the future, to discover patterns and anomalies, and to gain the insight necessary for making
faster and better decisions.
But before you can do any of those things, you need to ingest, store, transform, integrate, and
prepare your data. Guess what? You can do all of those things in Azure Synapse Analytics –
without having to write any code!
In this session, we will cover the fundamentals of data integration in Azure Synapse Analytics.
First, we will discuss when Azure Synapse Analytics is the right tool of choice. Then, we will go
through what Pipelines and Data Flows are, and when to use them. Finally, we will see how
easy it is to ingest and transform data both on-premises and in the cloud.

3. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net

4. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

5. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

6. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

7. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Store
Transform
Integrate
Prepare

8. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)

9. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse Analytics
What?
Pipelines & Data Flows
How?
Data Integration
When?
…the next 45 minutes…

10. Azure Synapse
Analytics

11. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
All-in-one platform for analytical projects:
• Data Lake (All Data)
• Data Warehouse (Relational Data)
• Data Analytics (SQL, Spark)
• Data Integration (Ingest, Transform)

12. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
Deeply integrated with other services:
• Azure Purview
• Azure Machine Learning
• Azure Cosmos DB
• Power BI

13. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Who can use Azure Synapse Analytics?
Built for collaboration between:
• Data Engineers
• Data Analysts
• Data Scientists
• Data Consumers

14. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Integration
Data

15. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

16. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Copy Data Transform Data

17. Pipelines &
Data Flows

18. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Pipelines?

19. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Activities?

20. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Datasets?

21. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Linked Services?

22. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flows?

23. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Triggers?

24. DEMO
DEMO
Pipelines &
Data Flows

25. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)

26. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

27. ?
?

28. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse or Azure Data Factory?

29. © 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/adf