Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Cloud-Native Streaming and Event-Driven Microservices

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Próximo SlideShare
TDD for Microservices
TDD for Microservices
Cargando en…3
×

Eche un vistazo a continuación

1 de 58 Anuncio

Cloud-Native Streaming and Event-Driven Microservices

Descargar para leer sin conexión

MARIUS BOGOEVICI SPRING CLOUD STREAM LEAD
Join us for an introduction to Spring Cloud Stream, a framework for creating event-driven microservices that builds on on the ease of development and execution of Spring Boot, the cloud-native capabilities of Spring Cloud, and the message-driven programming model of Spring Integration. See how Spring Cloud Stream’s abstractions and opinionated primitives allow you to easily build applications that can interchangeably use RabbitMQ, Kafka or Google PubSub without changing the application logic. Finally, we will show how these applications can be orchestrated and deployed on different modern runtimes such as Cloud Foundry, Kubernetes or Mesos using Spring Cloud Data Flow.

MARIUS BOGOEVICI SPRING CLOUD STREAM LEAD
Join us for an introduction to Spring Cloud Stream, a framework for creating event-driven microservices that builds on on the ease of development and execution of Spring Boot, the cloud-native capabilities of Spring Cloud, and the message-driven programming model of Spring Integration. See how Spring Cloud Stream’s abstractions and opinionated primitives allow you to easily build applications that can interchangeably use RabbitMQ, Kafka or Google PubSub without changing the application logic. Finally, we will show how these applications can be orchestrated and deployed on different modern runtimes such as Cloud Foundry, Kubernetes or Mesos using Spring Cloud Data Flow.

Anuncio
Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Cloud-Native Streaming and Event-Driven Microservices (20)

Anuncio

Más de VMware Tanzu (20)

Más reciente (20)

Anuncio

Cloud-Native Streaming and Event-Driven Microservices

  1. 1. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Streaming and Event-Driven Microservices Marius Bogoevici, Pivotal @mariusbogoevici
  2. 2. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who am I ? • Software Engineer with Pivotal • Spring ecosystem contributor since 2008: – Spring Integration, Spring XD, Spring Integration Kafka, – Spring Cloud Stream, Spring Cloud Data Flow • Co-author, “Spring Integration in Action”, Manning, 2012
  3. 3. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Streaming • Processing large quantities of data as fast as possible (near real-time) • Use cases: • real-time data movement (ETL) • predictive maintenance, fraud detection, IoS, QoS • enterprise integration meets Big Data • High throughput/low latency • Specific operations: grouping, ordering, windowing • Transition in message middleware design: • Traditional (JMS-based, RabbitMQ) • Log-style (Kafka, Amazon Kinesis, Azure Event Hubs) • Cloud-native: Google Pub Sub
  4. 4. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why microservices for data processing? • Cohesiveness around business capability (“do one thing and do it well”) • Organizational alignment (Conway’s Law), cross-team collaboration • Development agility • Optimized for replacement • Enable continuous delivery • Failure isolation • Granular resource tuning: • scaling out the critical parts of the pipeline • per-process: memory, CPU, network 4
  5. 5. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event-Driven/Messaging Microservices • Messaging via broker vs HTTP • Solving communication complexity for data processing • Decoupling: • Physical: discovery • Temporal: availability • Eventual consistency vs. shared stores/distributed transactions • especially over heterogenous resources • Pub-sub makes it easy to add new elements to the topology
  6. 6. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and Event Streaming: Conceptually Similar Data Streaming: ingestion, analytics Async interaction, event sourcing
  7. 7. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How about operational complexity? • Distributed systems are inherently complex, operating them even more so • Operational prerequisites: • Self-servicing and provisioning • elastic infrastructure • Monitoring • Self-healing • Rapid delivery • CI/CD, deployment pipeline • automation • a DevOps culture is required https://martinfowler.com/bliki/MicroservicePrerequisites.html 7
  8. 8. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 8
  9. 9. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud native applications and platforms • Target a platform that makes running apps reliable, transparent and boring • In-built resource management • Memory, CPU, networking • Elastic scaling • Monitoring and failover • Health, logging, metrics • Routing and load balancing • Rolling upgrades 9 Apache YARN Apache Mesos Kubernetes
  10. 10. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Monolith, the Platform and the Microservice(s) 10 Spring Cloud Stream2015 Spring XD Spring Cloud Data Flow Spring Cloud Task
  11. 11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11 file jms http Messaging middleware Platform cassandra count-words Cloud-native event-driven microservices Apache Kafka
  12. 12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservices framework • Goal: simplify writing messaging applications • Bias towards event/data streaming • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 12
  13. 13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 13 Application Core Spring Integration Spring Messaging Reactive APIs Kafka Streams inputs outputs
  14. 14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14
  15. 15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 15
  16. 16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 16 Application Messaging Middleware Binderchannel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties
  17. 17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 17 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental
  18. 18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 18 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
  19. 19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 19 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
  20. 20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive Programming via Reactor (since 1.1) 20 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux<String> process(@Input(“input”) Flux<String> s) {
 return f.map(s -> s.toUppercase()); }
 }
  21. 21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 21 SpringBootApplication
 @EnableBinding(Processor.class)
 public class WordCountApplication {
 
 @StreamListener
 @Output("output")
 public Flux<?> countWords(@Input("input") Flux<String> words) {
 return words.window(ofSeconds(5), ofSeconds(1))
 .flatMap(window -> window.groupBy(word -> word)
 .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
 .map(count -> new WordCount(group.key(), count))));
 } }
  22. 22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ [dumb] Pipes and [smart] filters 22 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds
  23. 23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 23 http raw-sensor-data averages averages top-n Calculator
  24. 24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 24 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub
  25. 25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 25
  26. 26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 26 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?
  27. 27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS
  28. 28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !
  29. 29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 28 Averages HDDSHDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other AveragesAverages consumers are competing within a group
  30. 30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 29
  31. 31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 30 Average calculator Average calculator http raw-sensor-data
  32. 32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 31 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data
  33. 33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 32 Average calculator Average calculator source partition 1 partition 2 raw-sensor-data Spring Cloud Stream: Partitioning
  34. 34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies: [relatively] easy to deploy … 33 http hdfs spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1
  35. 35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 34 http raw-sensor-data averages top-n Calculator Failure detector averagesaverages HDFSHDFSHDFS
  36. 36. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 35
  37. 37. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 36 Stream definition Spring Boot Apps built with Spring Cloud Stream httpfile = http | file |
  38. 38. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  39. 39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  40. 40. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  41. 41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  42. 42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38 cassandra gpfdist http stream1 = http | count-words | file stream2 = jms | cassandra Kafka Data Flow Server DB jms file jms http Message System Platform cassandra count-words
  43. 43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39
  44. 44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http
  45. 45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http LoadBalancer
  46. 46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work LoadBalancer
  47. 47. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  48. 48. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  49. 49. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  50. 50. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  51. 51. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  52. 52. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  53. 53. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  54. 54. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties
  55. 55. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties app.http.count=2 app.work.count=3 app.hdfs.count=4 app.http.producer.partitionKeyExpression=payload.id
  56. 56. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Resource Management 40 http http work work work app.work.spring.cloud.deployer.cloudfoundry.memory=2048
  57. 57. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some links … http://cloud.spring.io/spring-cloud-stream http://cloud.spring.io/spring-cloud-dataflow https://github.com/spring-cloud/spring-cloud-stream-samples
  58. 58. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ?

×