Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

1.718 visualizaciones

Publicado el

Understanding Azure Data Factory: The What, When, and Why (Presented at NIC on February 6th, 2020)

Publicado en: Datos y análisis
  • Sé el primero en comentar

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

  1. 1. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
  2. 2. Understanding Azure Data Factory The What, When, and Why Cathrine Wilhelmsen NIC · February 6th, 2019
  3. 3. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Understanding Azure Data Factory
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What? When? Why?
  8. 8. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  9. 9. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
  10. 10. Azure Data Factory
  11. 11. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  12. 12. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  13. 13. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Integration Runtimes Triggers Templates
  14. 14. DEMO Let's look inside Azure Data Factory!
  15. 15. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  16. 16. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What is the Copy Data Activity? * * Cathrine's opinion :)
  17. 17. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Process: Binary Files Source Sink
  18. 18. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
  19. 19. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
  20. 20. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
  21. 21. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
  22. 22. DEMO Let's copy some data!
  23. 23. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) what if my systems are on-premises?
  24. 24. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Hybrid Azure Data Factory
  25. 25. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What are Integration Runtimes?
  26. 26. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Azure Integration Runtime
  27. 27. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Self-Hosted Integration Runtime
  28. 28. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Copy Data Scenarios
  29. 29. DEMO Let's connect to an on-prem SQL Server!
  30. 30. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Ok, so we can copy data… Copy Data Transform Data
  31. 31. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) …what about transforming data? Copy Data Transform Data
  32. 32. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Mapping or Wrangling
  33. 33. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  34. 34. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) How do Mapping Data Flows work?
  35. 35. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) What are Wrangling Data Flows?
  36. 36. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) How do Wrangling Data Flows work?
  37. 37. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Mapping Data Flows Wrangling Data Flows
  38. 38. DEMO Let's transform some data!
  39. 39. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) how do we schedule data pipelines?
  40. 40. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Trigger pipelines…
  41. 41. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Triggers: Schedule
  42. 42. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Triggers: Tumbling Window
  43. 43. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Triggers: Event Based
  44. 44. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Triggers: Now
  45. 45. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Monitoring Triggers
  46. 46. DEMO Let's schedule some pipelines!
  47. 47. Azure Data Architectures
  48. 48. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Advanced Analytics on Big Data https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/
  49. 49. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Real-time Analytics https://azure.microsoft.com/en-us/solutions/architecture/real-time-analytics/
  50. 50. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Modern Data Warehouse https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/
  51. 51. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Ingest Azure Data Factory Serve Azure Synapse Analytics Visualize Power BI Store Azure Data Lake Storage Data Pipeline Orchestration and Monitoring Azure Data Factory Sources Cloud SaaS Prepare Wrangling Data Flows Transform Mapping Data Flows On-Premises
  52. 52. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Serve Azure Synapse Analytics Visualize Power BI Sources Cloud SaaS On-Premises
  53. 53. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  54. 54. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!

×