Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

dA Platform Overview

232 visualizaciones

Publicado el

dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

dA Platform Overview

  1. 1. Robert Metzger Patrick Lucas dA Platform Production-ready stream processing with Apache Flink®
  2. 2. © 2018 data Artisans2 WHY WE BUILD DA PLATFORM The need for streaming platforms The right tools from day one Operations for stateful streaming applications
  3. 3. © 2018 data Artisans3 THE NEED FOR STREAMING PLATFORMS DevOps model Integration with internal infrastructure Integrate once, scale adoption within organization Focus on use-cases instead of infrastructure • Logging • Metrics • State storage • Data access • Security • Auditing
  4. 4. © 2018 data Artisans4 THE RIGHT TOOLS FROM DAY ONE
  5. 5. © 2018 data Artisans5 STATEFUL STREAMING OPERATIONS Managing a stateful stream processor on stateless containers is challenging. STATEFUL STREAM PROCESSING Stateful stream processing is disrupting analytics and application development. KUBERNETES AND THE WORLD OF CONTAINERS Kubernetes offers easy to use container orchestration. But containers are stateless.
  6. 6. © 2018 data Artisans6 STATEFUL STREAMING OPERATIONS STATEFUL STREAM PROCESSING Stateful stream processing is disrupting analytics and application development. KUBERNETES AND THE WORLD OF CONTAINERS Kubernetes offers easy to use container orchestration. But containers are stateless.
  7. 7. © 2018 data Artisans7 WHY WE BUILD DA PLATFORM The need for streaming platforms Easy integration and focus on use-cases The right tools from day one Get started with stream processing in no time with a turnkey solution Operations for stateful streaming applications Bringing stateful streaming into the world of containers
  8. 8. © 2018 data Artisans8 HOW DOES IT WORK?
  9. 9. © 2018 data Artisans9 DECLARATIVE CONTROL OF DEPLOYMENTS Kubernetes Specify your Flink streaming deployment, dA Platform takes care of configuring, deploying and operating it. Status report / Event Log Log messages from Flink System Metrics Flink User Interface Deployment spec state: running job: fraud-detection.jar parallelism: 60 flinkConfiguration: {…} cpu: 8 …
  10. 10. © 2018 data Artisans10 DECLARATIVE CONTROL IN ACTION Deployment spec state: cancelled job: fraud-detection.jar parallelism: 60 flinkConfiguration: {…} cpu: 8 … In this case, the specification of a deployment changed to “cancelled”. The Flink cluster in Kubernetes will be deallocated. Kubernetes Changing the specification of a deployment will be reflected in the cluster.
  11. 11. © 2018 data Artisans11 DECLARATIVE CONTROL IN ACTION Deployment spec state: running job: fraud-detection.jar parallelism: 60 -> 80 flinkConfiguration: {…} cpu: 8 … The Flink cluster will scale up to the specified size. Kubernetes Changing the specification of a deployment will be reflected in the cluster. Scale up cluster
  12. 12. © 2018 data Artisans12 MANAGING STREAMING APPS AND STATE dA Platform allows to perform stateful deployment changes. Deployment changes: • Upgrading the Flink job • Upgrading Flink • Changing the parallelism • Changing resource allocations • Changing a configuration parameter • Suspending / resuming a deployment • … Scale up cluster Kubernetes Migrate StateMigrate state
  13. 13. © 2018 data Artisans13 VERSIONED APPLICATIONS, NOT JOBS/JARS STREAM PROCESSING APPLICATION Version 3 Version 2 Version 1 Code and Application State upgrade upgrade NEW APPLICATION Version 3a Version 2a fork / duplicate
  14. 14. © 2018 data Artisans14 IMPLEMENTATION
  15. 15. © 2018 data Artisans15 ARCHITECTURE Apache Flink Stateful stream processing Kubernetes Container platform Logging Metrics Real-time Analytics Anomaly- & Fraud Detection Real-time Data Integration Reactive Microservices dA Application Manager Application lifecycle management
  16. 16. © 2018 data Artisans16 ARCHITECTURE All components of dA Platform are shipped as Docker containers We provide an installer for setting up the platform on Kubernetes The containers for Flink follow the same versioning scheme as the Apache Flink releases. The dAP distribution Flink has minor patches: compatible with the open source releases. Metrics and logging components are provided for demonstration purposes. We recommend integration with existing systems.
  17. 17. © 2018 data Artisans17 APPLICATION MANAGER Application Manager is the central orchestration and lifecycle management component of dA platform. Resource Management Application Manager REST API CI/CD Integration Web interface Resource Allocation Job Control
  18. 18. © 2018 data Artisans18 APPLICATION MANAGER Rich, interactive user interface Based on the main REST API Deployment of Flink clusters on Kubernetes Control Flink jobs Generate configuration for metrics, logging, HA, state backups, security, … Trigger savepoints, cancel jobs, start jobs with savepoints, check Flink health, … Keep track of Flink jobs, configuration parameters, savepoints, events Stored on a persistent volume Declarative REST API For integration with existing systems such as CI/CD pipelines
  19. 19. © 2018 data Artisans19 HOOKS FOR CI/CD PIPELINES Kubernetes Cluster Application Version 1 CI Service dA Application Manager Trigger CI Call API Application Version 1 Stateful Upgrade Push Update Application Version 2
  20. 20. © 2018 data Artisans20 DEMO
  21. 21. © 2018 data Artisans21 WRAP-UP
  22. 22. © 2018 data Artisans22 AVAILABILITY • dA Platform has been announced at Flink Forward Berlin 2017, with a closed beta program ‒ 190 beta signups, successful collaboration with a large number of beta users • As of March 2018, dA Platform is generally available • Trial versions are available for download: data-artisans.com/download ‒ Recommendation: Download the trial virtual machine! • dA Platform license includes Enterprise Support for Apache Flink dA Platform Trial VM Download Now
  23. 23. © 2018 data Artisans23 ROADMAP Improved Access Control Introduce ownership for resources (deployments, savepoints, jobs), control access based on ownership. Metadata management Awareness for data sources such as Kafka topics, files in S3 or HDFS. SQL SQL will be a first class citizen of dA Platform with interfaces for interactive queries and integration with existing SQL tools. Multi-Datacenter fail-over Run dA Platform in redundant data centers, automatically moving the processing to a standby data center in case of a failure.
  24. 24. © 2018 data Artisans24 SUMMARY dA Platform is a turnkey solution for stateful stream processing with Apache Flink. Apache Flink Stateful stream processing Kubernetes Container platform Logging Metrics dA Application Manager Application lifecycle management It solves common challenges such as job upgrades, configuration changes or state migrations..
  25. 25. © 2018 data Artisans25 Q & A
  26. 26. © 2018 data Artisans26 data-artisans.com data-artisans.com/blog @dataArtisans data Artisans data-artisans STAY IN TOUCH

×