Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow and Micrometer - Christian Tzolov

376 visualizaciones

Publicado el

Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow and Micrometer by Christian Tzolov at SpringOne Tour 2019

Publicado en: Software
  • -- DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT -- ......................................................................................................................... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... (Unlimited)
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow and Micrometer - Christian Tzolov

  1. 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Christian Tzolov (@christzolov) March 2019 Real-time Analysis of Data Processing Pipelines Spring Cloud Data Flow & Micrometer
  2. 2. About @christzolov Pivotal engineer, Spring Cloud Data Flow team Apache Committer/PMC member Distributed, Data-intensive Systems and Toolkits
  3. 3. Agenda ■ Distributed, Data-Intensive Systems ■ Spring Cloud Data Flow toolkit ■ Operational metrics and monitoring ■ Micrometer, Time Series and Dimensions ■ Architectural Patterns and Practices ■ Q+A
  4. 4. “We call an application data-intensive if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing.”
  5. 5. What is Spring Cloud Data Flow A toolkit for building data integration, real- time streaming, and batch data processing pipelines.
  6. 6. What is Spring Cloud Data Flow A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler.
  7. 7. What is Spring Cloud Data Flow Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler. A toolkit for building data integration, real- time streaming, and batch data processing pipelines.
  8. 8. What is Spring Cloud Data Flow Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler. A toolkit for building data integration, real- time streaming, and batch data processing pipelines.
  9. 9. What is Spring Cloud Data Flow Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler. A toolkit for building data integration, real- time streaming, and batch data processing pipelines.
  10. 10. What is Spring Cloud Data Flow Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler. A toolkit for building data integration, real- time streaming, and batch data processing pipelines.
  11. 11. Runtime and Message Broker Abstraction Kubernetes Cloud Foundry Local / Dev Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Solace Opportunities: Same code; Same tests; Works with a variety of Message Brokers
  12. 12. Common Denominator = Spring Boot => Consolidate On Development Practices Test Infrastructure CI/CD Tooling and Automation Operational Metrics and Monitoring
  13. 13. Data Processing Pipeline
  14. 14. Data Flow Lifecycle • Data Flow accepts the data pipeline definition and delegates to Skipper for lifecycle managements. • Operational metrics and Monitoring.
  15. 15. Legacy Spring Cloud Metrics Collector
  16. 16. What we want Micrometer Time Series (Influx, Prometheus) Grafana
  17. 17. Time Series & Dimensional Data Model ● A time series is a series of data points ordered in time order. ● timestamped values belonging to the same metric and the same set of labeled dimensions. ● Every time series is uniquely identified by its metric name and a set of key-value pairs, known as labels, tags, dimensions.
  18. 18. Data Flow 2.0 Monitoring Architecture Support for Prometheus and InfluxDB
  19. 19. Data Pipeline Monitoring
  20. 20. Supported Metrics ● Spring Boot 2.0 - JVM, CPU, File descriptor… ● Spring Integration - Channel, Source, Handler ● Data Flow App Starters – Tags: stream name, app name, type, CF process_cpu_usage { application_guid="20036", application_name="log", application_type="sink", instance_index="0", stream_name="s3", instance="10.40.1.170:20036", job="scdf", } spring_integration_send_seconds_count { application_guid="20036", application_name="log", application_type="sink", stream_name="s3", instance_index="0", name="input", result="success", type="channel" exception="none", instance="10.40.1.170:20036", job="scdf", }
  21. 21. Pull-based TSDB (Prometheus) Implications: • Service Discovery • Security • Short Living Tasks (PushGateway)
  22. 22. Data Flow in Kubernetes Prometheus
  23. 23. Data Flow in PCF InfluxDB PromRegator
  24. 24. SCDF Monitoring with hosted Grafana and InfluxDB
  25. 25. Data Flow Analytics with Micrometer Counter Processor & Sink
  26. 26. Twitter Analytics
  27. 27. Next Steps ● Spring Cloud Stream Task support ● Monitor Data Flow and Skipper ● PromRegator and PCF improvements ● ?!
  28. 28. References ● Spring Cloud Data Flow – Stream Monitoring: http://docs.spring.io/spring-cloud- dataflow/docs/2.1.0.BUILD-SNAPSHOT/reference/htmlsingle/#streams-monitoring ● Twitter Analytics Sample: https://docs.spring.io/spring-cloud-dataflow- samples/docs/current/reference/htmlsingle/#spring-cloud-data-flow-samples-twitter-analytics- overview ● Micrometer: https://micrometer.io ● Spring Boot Micrometer : https://docs.spring.io/spring- boot/docs/2.1.3.RELEASE/reference/htmlsingle/#production-ready-metrics ● Spring Integration – Micrometer: https://docs.spring.io/spring- integration/docs/5.1.2.RELEASE/reference/html/system-management-chapter.html#micrometer- integration ● Grafana: https://grafana.com ● Prometheus: https://prometheus.io
  29. 29. Q&A SCDF STREAM MONITORING

×