SlideShare a Scribd company logo
1 of 18
Your SQL Man (I) Pvt Ltd. 1
Azure Data Factory
Your SQL Man (I) Pvt Ltd. 2
Overview
•Data Integration Service
•Data movement
•Transformation
•Both On-Premises & Cloud
Your SQL Man (I) Pvt Ltd. 3
Overview
Your SQL Man (I) Pvt Ltd. 4
Key Concepts
Your SQL Man (I) Pvt Ltd. 5
Key Concepts
• Datasets
• Datasets are named references/pointers to the data you want to use as an input or an output of an Activity. Datasets
identify data structures within different data stores including tables, files, folders, and documents.
• Activities
• Activities define the actions to perform on your data. Each activity takes zero or more datasetsas inputs and produces
one or more datasets as outputs. An activity is a unit of orchestration in Azure Data Factory. For example, you may
use a Copy activity to orchestrate copying data from one dataset to another.
• Pipelines
• Pipelines are a logical grouping of Activities. They are used to group activities into a unit that together perform a task.
For example, a sequence of several transformation Activities might be needed to cleanse log file data. This sequence
could have a complex schedule and dependencies that need to be orchestrated and automated. All of these activities
could be grouped into a single Pipeline named “CleanLogFiles”.
• Linked service
• Linked services define the information needed for Data Factory to connect to external resources. Linked services are
used for two purposes in Data Factory:
• To represent a data store including, but not limited to, an on-premises SQL Server, Oracle DB, File share or an Azure
Blob Storage account. As discussed above, Datasets represent the structures within the data stores connected to Data
Factory through a Linked service.
Your SQL Man (I) Pvt Ltd. 6
Supported Regions
•West US
•North Europe
Your SQL Man (I) Pvt Ltd. 7
Pricing
Your SQL Man (I) Pvt Ltd. 8
Datasets in Data Factory
•Input / Output
•Linked Service
•Data Structure
Your SQL Man (I) Pvt Ltd. 9
Your SQL Man (I) Pvt Ltd. 10
Your SQL Man (I) Pvt Ltd. 11
Pipelines & Activity
•Pipeline is a logical grouping of
Activities.
Your SQL Man (I) Pvt Ltd. 12
Data Stores
•Supported Data Stores
Your SQL Man (I) Pvt Ltd. 13
Gateway for On-Premises
•Install on your local
•Register using key
•Linked Service using Gateway
Your SQL Man (I) Pvt Ltd. 14
Copy Activity
•GUI Based
•Create all stuff in wizard
Your SQL Man (I) Pvt Ltd. 15
Transformation Activity
•HDInsight Entities
Your SQL Man (I) Pvt Ltd. 16
Monitor
•Monitoring States
Your SQL Man (I) Pvt Ltd. 17
Monitor
•Dataset State Diagram
Your SQL Man (I) Pvt Ltd. 18
Demo
Data factory

More Related Content

What's hot

What's hot (20)

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
 
Azure data factory
Azure data factoryAzure data factory
Azure data factory
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data Factory
 
Azure Data Factory v2
Azure Data Factory v2Azure Data Factory v2
Azure Data Factory v2
 
Core Concepts in azure data factory
Core Concepts in azure data factoryCore Concepts in azure data factory
Core Concepts in azure data factory
 
Microsoft Azure Data Factory Hands-On Lab Overview Slides
Microsoft Azure Data Factory Hands-On Lab Overview SlidesMicrosoft Azure Data Factory Hands-On Lab Overview Slides
Microsoft Azure Data Factory Hands-On Lab Overview Slides
 
Intro to Azure Data Factory v1
Intro to Azure Data Factory v1Intro to Azure Data Factory v1
Intro to Azure Data Factory v1
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Azure Data Factory Data Flows Training (Sept 2020 Update)
Azure Data Factory Data Flows Training (Sept 2020 Update)Azure Data Factory Data Flows Training (Sept 2020 Update)
Azure Data Factory Data Flows Training (Sept 2020 Update)
 
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
 
Azure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the CloudAzure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the Cloud
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
 
Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005
 
Azure data bricks by Eugene Polonichko
Azure data bricks by Eugene PolonichkoAzure data bricks by Eugene Polonichko
Azure data bricks by Eugene Polonichko
 
Data Lake: A simple introduction
Data Lake: A simple introductionData Lake: A simple introduction
Data Lake: A simple introduction
 
Azure Data Factory for Azure Data Week
Azure Data Factory for Azure Data WeekAzure Data Factory for Azure Data Week
Azure Data Factory for Azure Data Week
 
Modern Data Warehouse with Azure Synapse.pdf
Modern Data Warehouse with Azure Synapse.pdfModern Data Warehouse with Azure Synapse.pdf
Modern Data Warehouse with Azure Synapse.pdf
 
Azure DataBricks for Data Engineering by Eugene Polonichko
Azure DataBricks for Data Engineering by Eugene PolonichkoAzure DataBricks for Data Engineering by Eugene Polonichko
Azure DataBricks for Data Engineering by Eugene Polonichko
 

Similar to Azure Data Factory

Less01 db architecture
Less01 db architectureLess01 db architecture
Less01 db architecture
Imran Ali
 
ETL processes , Datawarehouse and Datamarts.pptx
ETL processes , Datawarehouse and Datamarts.pptxETL processes , Datawarehouse and Datamarts.pptx
ETL processes , Datawarehouse and Datamarts.pptx
ParnalSatle
 
Inb343 week2 sql server intro
Inb343 week2 sql server introInb343 week2 sql server intro
Inb343 week2 sql server intro
Fredlive503
 
A introduction to oracle data integrator
A introduction to oracle data integratorA introduction to oracle data integrator
A introduction to oracle data integrator
chkamal
 
Sql server 2008 r2 performance and scale
Sql server 2008 r2 performance and scaleSql server 2008 r2 performance and scale
Sql server 2008 r2 performance and scale
Klaudiia Jacome
 

Similar to Azure Data Factory (20)

Transform your data with Azure Data factory
Transform your data with Azure Data factoryTransform your data with Azure Data factory
Transform your data with Azure Data factory
 
Test labs 2016. Тестирование data warehouse
Test labs 2016. Тестирование data warehouse Test labs 2016. Тестирование data warehouse
Test labs 2016. Тестирование data warehouse
 
A Review of Data Access Optimization Techniques in a Distributed Database Man...
A Review of Data Access Optimization Techniques in a Distributed Database Man...A Review of Data Access Optimization Techniques in a Distributed Database Man...
A Review of Data Access Optimization Techniques in a Distributed Database Man...
 
A Review of Data Access Optimization Techniques in a Distributed Database Man...
A Review of Data Access Optimization Techniques in a Distributed Database Man...A Review of Data Access Optimization Techniques in a Distributed Database Man...
A Review of Data Access Optimization Techniques in a Distributed Database Man...
 
Less01 db architecture
Less01 db architectureLess01 db architecture
Less01 db architecture
 
The Real Scoop on Migrating from Oracle Databases
The Real Scoop on Migrating from Oracle DatabasesThe Real Scoop on Migrating from Oracle Databases
The Real Scoop on Migrating from Oracle Databases
 
ETL processes , Datawarehouse and Datamarts.pptx
ETL processes , Datawarehouse and Datamarts.pptxETL processes , Datawarehouse and Datamarts.pptx
ETL processes , Datawarehouse and Datamarts.pptx
 
Inb343 week2 sql server intro
Inb343 week2 sql server introInb343 week2 sql server intro
Inb343 week2 sql server intro
 
A introduction to oracle data integrator
A introduction to oracle data integratorA introduction to oracle data integrator
A introduction to oracle data integrator
 
Informatica_ Basics_Demo_9.6.ppt
Informatica_ Basics_Demo_9.6.pptInformatica_ Basics_Demo_9.6.ppt
Informatica_ Basics_Demo_9.6.ppt
 
Real world business workflow with SharePoint designer 2013
Real world business workflow with SharePoint designer 2013Real world business workflow with SharePoint designer 2013
Real world business workflow with SharePoint designer 2013
 
Sql server 2008 r2 performance and scale
Sql server 2008 r2 performance and scaleSql server 2008 r2 performance and scale
Sql server 2008 r2 performance and scale
 
Oracle
OracleOracle
Oracle
 
Migrating Data Warehouse Solutions from Oracle to non-Oracle Databases
Migrating Data Warehouse Solutions from Oracle to non-Oracle DatabasesMigrating Data Warehouse Solutions from Oracle to non-Oracle Databases
Migrating Data Warehouse Solutions from Oracle to non-Oracle Databases
 
A lap around Azure Data Factory
A lap around Azure Data FactoryA lap around Azure Data Factory
A lap around Azure Data Factory
 
UNIT 2 DATA WAREHOUSING AND DATA MINING PRESENTATION.pptx
UNIT 2 DATA WAREHOUSING AND DATA MINING PRESENTATION.pptxUNIT 2 DATA WAREHOUSING AND DATA MINING PRESENTATION.pptx
UNIT 2 DATA WAREHOUSING AND DATA MINING PRESENTATION.pptx
 
Building Data Warehouse in SQL Server
Building Data Warehouse in SQL ServerBuilding Data Warehouse in SQL Server
Building Data Warehouse in SQL Server
 
Module 1_Data Warehousing Fundamentals.pptx
Module 1_Data Warehousing Fundamentals.pptxModule 1_Data Warehousing Fundamentals.pptx
Module 1_Data Warehousing Fundamentals.pptx
 
Azure Data Engineer Course | Azure Data Engineer Training Hyderabad.pptx
Azure Data Engineer Course | Azure Data Engineer Training Hyderabad.pptxAzure Data Engineer Course | Azure Data Engineer Training Hyderabad.pptx
Azure Data Engineer Course | Azure Data Engineer Training Hyderabad.pptx
 
In-memory ColumnStore Index
In-memory ColumnStore IndexIn-memory ColumnStore Index
In-memory ColumnStore Index
 

More from HARIHARAN R

More from HARIHARAN R (16)

Power BI Admin Features & REST API
Power BI Admin Features & REST APIPower BI Admin Features & REST API
Power BI Admin Features & REST API
 
Understanding Power BI Data Model
Understanding Power BI Data ModelUnderstanding Power BI Data Model
Understanding Power BI Data Model
 
DAX - 101
DAX - 101 DAX - 101
DAX - 101
 
Power BI Publisher
Power BI Publisher Power BI Publisher
Power BI Publisher
 
Integrating flow and power apps with power bi
Integrating flow and power apps with power biIntegrating flow and power apps with power bi
Integrating flow and power apps with power bi
 
Latest Features in Power BI
Latest Features in Power BILatest Features in Power BI
Latest Features in Power BI
 
Real time analytics with Power BI
Real time analytics with Power BIReal time analytics with Power BI
Real time analytics with Power BI
 
A Deep Dive into to Power BI - level 2
A Deep Dive into to Power BI - level 2A Deep Dive into to Power BI - level 2
A Deep Dive into to Power BI - level 2
 
SQL Server Reporting Services 2016 Features
SQL Server Reporting Services 2016 FeaturesSQL Server Reporting Services 2016 Features
SQL Server Reporting Services 2016 Features
 
CPBIG - A Deep Dive into Power BI
CPBIG - A Deep Dive into Power BICPBIG - A Deep Dive into Power BI
CPBIG - A Deep Dive into Power BI
 
Azure Stream Analytics - Webinar
Azure Stream Analytics - WebinarAzure Stream Analytics - Webinar
Azure Stream Analytics - Webinar
 
Azure Logic, API & Mobile Apps
Azure Logic, API & Mobile AppsAzure Logic, API & Mobile Apps
Azure Logic, API & Mobile Apps
 
Introduction to Power BI
Introduction to Power BIIntroduction to Power BI
Introduction to Power BI
 
Azure search
Azure searchAzure search
Azure search
 
Analytics with R in SQL Server 2016
Analytics with R in SQL Server 2016Analytics with R in SQL Server 2016
Analytics with R in SQL Server 2016
 
Azure catalog
Azure catalogAzure catalog
Azure catalog
 

Recently uploaded

Salient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functionsSalient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functions
KarakKing
 
Spellings Wk 3 English CAPS CARES Please Practise
Spellings Wk 3 English CAPS CARES Please PractiseSpellings Wk 3 English CAPS CARES Please Practise
Spellings Wk 3 English CAPS CARES Please Practise
AnaAcapella
 
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
ZurliaSoop
 

Recently uploaded (20)

ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
 
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
 
Food safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdfFood safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdf
 
Salient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functionsSalient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functions
 
REMIFENTANIL: An Ultra short acting opioid.pptx
REMIFENTANIL: An Ultra short acting opioid.pptxREMIFENTANIL: An Ultra short acting opioid.pptx
REMIFENTANIL: An Ultra short acting opioid.pptx
 
Sensory_Experience_and_Emotional_Resonance_in_Gabriel_Okaras_The_Piano_and_Th...
Sensory_Experience_and_Emotional_Resonance_in_Gabriel_Okaras_The_Piano_and_Th...Sensory_Experience_and_Emotional_Resonance_in_Gabriel_Okaras_The_Piano_and_Th...
Sensory_Experience_and_Emotional_Resonance_in_Gabriel_Okaras_The_Piano_and_Th...
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
General Principles of Intellectual Property: Concepts of Intellectual Proper...
General Principles of Intellectual Property: Concepts of Intellectual  Proper...General Principles of Intellectual Property: Concepts of Intellectual  Proper...
General Principles of Intellectual Property: Concepts of Intellectual Proper...
 
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdfUGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
 
Graduate Outcomes Presentation Slides - English
Graduate Outcomes Presentation Slides - EnglishGraduate Outcomes Presentation Slides - English
Graduate Outcomes Presentation Slides - English
 
How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17
 
Micro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdfMicro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdf
 
Mehran University Newsletter Vol-X, Issue-I, 2024
Mehran University Newsletter Vol-X, Issue-I, 2024Mehran University Newsletter Vol-X, Issue-I, 2024
Mehran University Newsletter Vol-X, Issue-I, 2024
 
Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)
 
How to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POSHow to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POS
 
Spellings Wk 3 English CAPS CARES Please Practise
Spellings Wk 3 English CAPS CARES Please PractiseSpellings Wk 3 English CAPS CARES Please Practise
Spellings Wk 3 English CAPS CARES Please Practise
 
Application orientated numerical on hev.ppt
Application orientated numerical on hev.pptApplication orientated numerical on hev.ppt
Application orientated numerical on hev.ppt
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 
Spatium Project Simulation student brief
Spatium Project Simulation student briefSpatium Project Simulation student brief
Spatium Project Simulation student brief
 
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
 

Azure Data Factory

  • 1. Your SQL Man (I) Pvt Ltd. 1 Azure Data Factory
  • 2. Your SQL Man (I) Pvt Ltd. 2 Overview •Data Integration Service •Data movement •Transformation •Both On-Premises & Cloud
  • 3. Your SQL Man (I) Pvt Ltd. 3 Overview
  • 4. Your SQL Man (I) Pvt Ltd. 4 Key Concepts
  • 5. Your SQL Man (I) Pvt Ltd. 5 Key Concepts • Datasets • Datasets are named references/pointers to the data you want to use as an input or an output of an Activity. Datasets identify data structures within different data stores including tables, files, folders, and documents. • Activities • Activities define the actions to perform on your data. Each activity takes zero or more datasetsas inputs and produces one or more datasets as outputs. An activity is a unit of orchestration in Azure Data Factory. For example, you may use a Copy activity to orchestrate copying data from one dataset to another. • Pipelines • Pipelines are a logical grouping of Activities. They are used to group activities into a unit that together perform a task. For example, a sequence of several transformation Activities might be needed to cleanse log file data. This sequence could have a complex schedule and dependencies that need to be orchestrated and automated. All of these activities could be grouped into a single Pipeline named “CleanLogFiles”. • Linked service • Linked services define the information needed for Data Factory to connect to external resources. Linked services are used for two purposes in Data Factory: • To represent a data store including, but not limited to, an on-premises SQL Server, Oracle DB, File share or an Azure Blob Storage account. As discussed above, Datasets represent the structures within the data stores connected to Data Factory through a Linked service.
  • 6. Your SQL Man (I) Pvt Ltd. 6 Supported Regions •West US •North Europe
  • 7. Your SQL Man (I) Pvt Ltd. 7 Pricing
  • 8. Your SQL Man (I) Pvt Ltd. 8 Datasets in Data Factory •Input / Output •Linked Service •Data Structure
  • 9. Your SQL Man (I) Pvt Ltd. 9
  • 10. Your SQL Man (I) Pvt Ltd. 10
  • 11. Your SQL Man (I) Pvt Ltd. 11 Pipelines & Activity •Pipeline is a logical grouping of Activities.
  • 12. Your SQL Man (I) Pvt Ltd. 12 Data Stores •Supported Data Stores
  • 13. Your SQL Man (I) Pvt Ltd. 13 Gateway for On-Premises •Install on your local •Register using key •Linked Service using Gateway
  • 14. Your SQL Man (I) Pvt Ltd. 14 Copy Activity •GUI Based •Create all stuff in wizard
  • 15. Your SQL Man (I) Pvt Ltd. 15 Transformation Activity •HDInsight Entities
  • 16. Your SQL Man (I) Pvt Ltd. 16 Monitor •Monitoring States
  • 17. Your SQL Man (I) Pvt Ltd. 17 Monitor •Dataset State Diagram
  • 18. Your SQL Man (I) Pvt Ltd. 18 Demo Data factory