a talk about azure synapse aimed to help people who are not data experts understand what synapse is and how you can integrate it with other technologies

1. Azure Synapse
For mere mortals

2. About me
• Microsoft MVP 12 years
• Freelance Cloud Architect
• Based in Newcastle, UK
• https://mikestephenson.me/
• @Michael_Stephen

3. Trying to understand Synapse

4. What is Synapse

5. Reference Architecture 1

6. Reference Architecture 2

7. Reference Architecture 3

8. Trying to make sense of it all

9. What is Synapse
Serverless SQL Dedicated SQL Spark Pool Pipelines Synapse Studio
What Serverless SQL on demand Formerly SQL Data
Warehouse
[Optional]
Serverless Apache Spark
Adds additional features
to let you use Apache
Spark more closely in
Synapse. EG AAD,
AzureML
Like Data Factory
Low code ETL / ELT
Developer studio for
building synapse
solutions
Cost Pay per query Pay per hour Pay per Free
Use case - Ad-hoc and occasional queries
- Experimentation
- Data science
experiments
- Batch jobs
- Integration of data in
and out of Synapse

10. Synapse by Example

11. Scenario 1: Query Data from Data Lake w/ Serverless SQL
Data Platform
Landing
Data Lake
Storage
Serverless
SQL
Experiment

12. Real World Complex Example
Data Platform
Landing Transform Raw/Curated
Orchestrate
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Data Lake
Storage
Synapse Pipelines
ELT
Synapse Pipelines
Serverless SQL
Power BI
Logic Apps
Serve
Experiment / Analyze
Integrate
Dedicated SQL

13. Scenario 2: Cosmos DB & Synapse Link
Data Platform
Application Platform
Synapse Link
Cosmos Serverless
SQL

14. Real World Complex Example
Data Platform
Application Platform
Synapse Link
Cosmos Serverless
SQL
Landing Transform Raw/Curated
Export
Orchestrate
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Synapse Pipelines
Data Lake
Storage
Synapse Pipelines
Serve
Power BI
Dedicated
SQL
ELT
Synapse Pipelines

15. Scenario 3: Dataverse & Synapse Link
Data Platform
Landing
Data Lake
Storage
Power Platform and CRM
CRM
Synapse Link
Portals
Dataverse
Power Apps
Serverless
SQL
Experiment

16. Real World Complex Example
Data Platform
Landing Transform Raw/Curated
Export
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Synapse Pipelines
Data Lake
Storage
Power Platform and CRM
CRM
Synapse Link
Portals
Dataverse
Power Apps
Serve
Power BI
Dedicated
SQL

17. Scenario 4: Processing Event Data from Event Hub
Data Platform
Landing
Data Lake
Storage
Event Hub Event Hub
Capture
Logic App
Test Simulator
Serverless
SQL
Experiment

18. Real World Complex Example
Data Platform
Landing Transform Raw/Curated
Orchestrate
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Data Lake
Storage
Synapse Pipelines
Event Hub Event Hub
Capture
Various App
Scenarios
Dedicated
SQL
Export
Synapse Pipelines
Serve
Power BI

19. Real-World Architecture

20. Gen2 DataLake
SQL Azure
Purview
Power BI
Synapse Data
Warehouse
Storage / Data Warehouse / Analytics
ETL & ELT
Data Factory
Synapse
Pipelines
Security / Management / Operations
Azure Active
Directory
Data Visualization
& Analysis
Power Apps
Dataverse
Systems of Engagement
Data Platform
Synapse Spark
pool
Event Data
Transactional Data
Event Hub
Cosmos DB
Synapse
Serverless
SQL Pool
DNS Zones
Virtual Network
Private Link
Synapse Private
Link Hub
Private Endpoints
Network Security
Groups
Log Analytics Key Vault Security Centre Cost Analysis
Helper Functions
& Automation
Infrastructure
Logic Apps
Azure Functions
Excel
Notebooks
Application Lifecycle
Pipelines Testing
Repos
Boards

22. Lessons Learnt

23. Lessons Learnt
• Cost Management – See next slide
• You can get value quickly
• It can get complicated
• Its powerful
• Especially when linked with other Azure Features
• Loads of deployment options
• Loads of security features and options

24. Cost
Costs to be cautious of:
• Dedicated SQL Pool
• Event Hub Capture
• Storage accrues
• Log Data
MSDN for learning
Non-Production Environment
£800 p/m
Smallish Production-like Environment
£2.5k p/m

25. Getting Started
• Its viable to start in MSDN
• These Examples hopefully can help you get started
• More Info:
• https://www.mikestephenson.me/2021/09/01/pausing-synapse-dedicated-sql-pool-with-devops-pipeline/
• https://www.mikestephenson.me/2021/08/08/trying-out-event-hub-capture-to-synapse-part-1/
• https://www.mikestephenson.me/2021/08/09/trying-out-event-hub-capture-to-synapse-part-2/
• https://www.mikestephenson.me/2021/08/05/playing-around-with-synapse-link-for-cosmosdb/
• https://www.mikestephenson.me/2021/08/03/query-synapse-serverless-sql-from-logic-apps/

26. Questions