Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

2.249 visualizaciones

Publicado el

Pipelines and Packages: Introduction to Azure Data Factory (Presented at DATA:Scotland on September 13th, 2019)

Publicado en: Datos y análisis
  • Inicia sesión para ver los comentarios

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

  1. 1. Pipelines and Packages: Introduction to Azure Data Factory Cathrine Wilhelmsen DATA:Scotland · September 13th, 2019
  2. 2. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  3. 3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Pipelines and Packages: Introduction to Azure Data Factory
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  9. 9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) 10 years ago…
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS SQL Server Integration Services
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Then…
  13. 13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v1 Azure Data Factory Version 1
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  15. 15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Today…
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v2 Azure Data Factory Version 2
  18. 18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why? How? What?
  20. 20. Azure Data Factory
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  22. 22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Data Flows Templates Triggers
  24. 24. DEMO Let's look inside Azure Data Factory!
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Wait… I already have thousands of SSIS packages!
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) And… You told me to use this Biml thing!
  27. 27. SSIS Lift and Shift
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What does Lift and Shift mean?
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) * Why should you Lift and Shift SSIS?
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Azure-SSIS Integration Runtime like all VMs
  33. 33. DEMO Let’s lift and shift some SSIS Packages!
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ≈ ≈ ≈ ≈ ≈ ≈ ADF vs SSIS
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
  36. 36. Mapping Data Flows
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do Mapping Data Flows work?
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Mapping Data Flows?
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is Schema Drift?
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in SSIS
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in ADF
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Oh no!
  44. 44. DEMO Let's transform some data!
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lessons Learned
  46. 46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

×