Azure Data Platform Overview - Azure Data Factory - Lambda Architecture

1. January 2019

2. TrustedProductive IntelligentHybrid
Azure. Cloud for all.

3. Trusted
Intelligent
Hybrid
Productive
HIPAA /
HITECH Act
FERPAGxP
21 CFR Part 11
ISO 27001 SOC 1 Type 2ISO 27018
CSA STAR
Self-Assessment
Singapore
MTCS
UK
G-Cloud
Australia
IRAP/CCSL
FISC Japan
New Zealand
GCIO
China
GB 18030
EU
Model Clauses
ENISA
IAF
Argentina
PDPA
Japan CS
Mark Gold
CDSA Shared
Assessments
Japan My
Number Act
FACT UK
GLBA
Spain
ENS
PCI DSS
Level 1
MARS-E FFIEC
China
TRUCS
SOC 2 Type 2 SOC 3
Canada
Privacy Laws
MPAA
Privacy
Shield
ISO 22301
India
MeitY
Germany IT
Grundschutz
workbook
Spain
DPA
CSA STAR
Certification
CSA STAR
Attestation
HITRUST IG Toolkit UK
China
DJCP
ISO 27017
GLOBALINDUSTRYREGIONAL
More certifications than any cloud provider

4. 50+Azure
regions
Trusted
Intelligent
Hybrid
Productive
Learn more:
Microsoft.com/datacenter

5. © Microsoft Corporation
Open Source Support
Applications
Infrastructure
Management
Databases &
middleware
App frameworks
& tools
DevOps

7. L A M B D A A R C H I T E C T U R E

8. M I C R O S O F T B I G D A T A B U I L D I N G B L O C K S
Intelligence
Dashboards &
Visualizations
Big Data Stores Machine Learning
and Analytics
Event Hubs
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Intelligence Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Bot
Framework
Azure SQL Data
Warehouse
Data Factory
Machine
Learning
Data Lake Store
Cognitive
Services
Power BI
Data
Sources
Apps
Sensors
and devices
Data
Information
Management
Azure SQL DB Azure
Databricks
IoT Hub

10. Data orchestration As-a-service
Security and compliance
Fully-managed data integration service in the cloud
A Z U R E D ATA F A C T O R Y
Hybrid Data movement
Data integration
Flexible

11. Modernize your data warehouse with Azure
big data and advanced analytics services such
as HDInsight and Data lake Analytics
Build custom data-driven SaaS applications
unique to your customer data using your
language of choice
Bring together all your sources of data to
understand your customers and drive
impactful business decisions
Flexible
Data integration

12. Orchestrate your data pipeline wherever your
data lives – in cloud or in self-hosted
environment
Meet your security and compliance needs
while taking advantage of truly hybrid
integration capabilities
Execute your SQL Server Integration Services
(SSIS) packages in the cloud
Hybrid
Data orchestration

13. Accelerate integration with managed data
movement as-a-service
30+ natively supported connectors across 18
global points of presence
Elastic data movement at scale
Serverless data movement with no
infrastructure to manage
Data movement
As-a-service

14. HYBRID DATA INTEGRATION AT SCALE
Any BI tool
Advanced
Analytics
Any language
Data Processing & MovementRelational data
Dashboards | Reporting
Mobile BI | Cubes
Machine Learning
Stream analytics Cognitive | AI
.NET | Java | R | Python
Ruby | PHP | Scala
Non-relational data
OLTP ERP CRM LOB
Social media DevicesWeb Media
CLOUD
ON-PREMISE
V-NET
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING

15. Azure Data Factory: How Does it Work?
Connect
& Collect
Transform
& Enrich
Publish Monitor

16. Azure Data Factory: Top-Level Concepts

17. Azure Data Factory: Top-Level Concepts

19. ADF Integration Runtime (IR) ▪ Integration Runtime is the compute
infrastructure used by ADF to provide data
integration capabilities such as:
▪ Activity dispatch & monitoring
▪ Data movement
▪ SSIS package execution
▪ Integration Runtime Types:
▪ Azure
▪ Azure-SSIS
▪ Self-Hosted
▪ IR provides a consistent provision and
monitoring experience across the network
environments
Portal Application & SDK
Azure Data Factory Service
Data Movement & Activity
Dispatch on-prem, Cloud,
VNET
Data Movement & Activity
Dispatch In Azure Public
Network, SSIS
VNET coming soon
Self-Hosted IR Azure IR

21. SSIS as an Integration Runtime
Execute/Manage
Provision
SSDT SSMS
Cloud
On-Premises
SSIS Server
Azure Data Factory V2
HDI
ML
 Provision SSIS IR via Data Factory
 Use SQL Server Data Tools (SSDT) to
author/deploy SSIS packages
 Use SQL Server Management Studio
(SSMS) to execute/manage SSIS
packages
 For those who want to move all/part
of their on-premises workloads and
“lift & shift” existing packages to Azure
SSIS IR on
ADF compute
resource

23. Branching Activities

24. Execute SSIS Packages Using Azure SSIS IR

26. Hybrid Data Integration Pattern 1
Analyze blog comments
Azure SQL Database
SQL Server
(on-premises)
Data Management
Gateway Req’d for ADF
Azure Data Factory
Capture blog comments via API
Drop into
Blob Store
Copy & lookup
Power BI Dashboard
Visualize and analyze
Transform via SPROC (ELT)
Transform via Dataflow (ETL)

27. Hybrid Data Integration Pattern 2
Data-Driven SaaS App: Sentiment Analysis w/Machine Learning
Azure Data Factory
Custom App
Blob Storage
Azure Functions

29. ADF Resources
Integration Runtime in Azure Data Factory
Provision the Azure-SSIS Integration Runtime in ADF
Create and Configure a Self-Hosted Integration Runtime
Azure Data Factory V2 Data Flow
Incremental/Delta Data Copy in ADF
Branching and Chaining Activities in an ADF Pipeline
Run an SSIS Package in ADF with the “Execute SSIS Package” Activity
Deploy an SSIS Package to an Azure SQL DB from SSMS
Azure Data Factory Pricing