Cloud-Native Streaming and Event-Driven Microservices

1. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Streaming and Event-Driven Microservices Marius Bogoevici, Pivotal @mariusbogoevici

2. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who am I ? • Software Engineer with Pivotal • Spring ecosystem contributor since 2008: – Spring Integration, Spring XD, Spring Integration Kafka, – Spring Cloud Stream, Spring Cloud Data Flow • Co-author, “Spring Integration in Action”, Manning, 2012

3. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Streaming • Processing large quantities of data as fast as possible (near real-time) • Use cases: • real-time data movement (ETL) • predictive maintenance, fraud detection, IoS, QoS • enterprise integration meets Big Data • High throughput/low latency • Specific operations: grouping, ordering, windowing • Transition in message middleware design: • Traditional (JMS-based, RabbitMQ) • Log-style (Kafka, Amazon Kinesis, Azure Event Hubs) • Cloud-native: Google Pub Sub

4. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why microservices for data processing? • Cohesiveness around business capability (“do one thing and do it well”) • Organizational alignment (Conway’s Law), cross-team collaboration • Development agility • Optimized for replacement • Enable continuous delivery • Failure isolation • Granular resource tuning: • scaling out the critical parts of the pipeline • per-process: memory, CPU, network 4

5. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event-Driven/Messaging Microservices • Messaging via broker vs HTTP • Solving communication complexity for data processing • Decoupling: • Physical: discovery • Temporal: availability • Eventual consistency vs. shared stores/distributed transactions • especially over heterogenous resources • Pub-sub makes it easy to add new elements to the topology

6. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and Event Streaming: Conceptually Similar Data Streaming: ingestion, analytics Async interaction, event sourcing

7. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How about operational complexity? • Distributed systems are inherently complex, operating them even more so • Operational prerequisites: • Self-servicing and provisioning • elastic infrastructure • Monitoring • Self-healing • Rapid delivery • CI/CD, deployment pipeline • automation • a DevOps culture is required https://martinfowler.com/bliki/MicroservicePrerequisites.html 7

8. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 8

9. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud native applications and platforms • Target a platform that makes running apps reliable, transparent and boring • In-built resource management • Memory, CPU, networking • Elastic scaling • Monitoring and failover • Health, logging, metrics • Routing and load balancing • Rolling upgrades 9 Apache YARN Apache Mesos Kubernetes

10. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Monolith, the Platform and the Microservice(s) 10 Spring Cloud Stream2015 Spring XD Spring Cloud Data Flow Spring Cloud Task

11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11 file jms http Messaging middleware Platform cassandra count-words Cloud-native event-driven microservices Apache Kafka

12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservices framework • Goal: simplify writing messaging applications • Bias towards event/data streaming • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 12

13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 13 Application Core Spring Integration Spring Messaging Reactive APIs Kafka Streams inputs outputs

14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14

16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 16 Application Messaging Middleware Binderchannel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties

17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 17 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental

18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 18 @SpringBootApplication  @EnableBinding(Processor.class)  public class UppercaseProcessor {    @StreamListener(“input”)  @SendTo(“output”)  public String process(String s) {  return s.toUpperCase();  }  }

19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 19 @SpringBootApplication  @EnableBinding(Processor.class)  public class UppercaseProcessor {    @StreamListener(“input”)  @SendTo(“output”)  public String process(String s) {  return s.toUpperCase();  }  }

20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive Programming via Reactor (since 1.1) 20 @SpringBootApplication  @EnableBinding(Processor.class)  public class UppercaseProcessor {    @StreamListener  @Output(“output”)  public Flux<String> process(@Input(“input”) Flux<String> s) {  return f.map(s -> s.toUppercase()); }  }

21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 21 SpringBootApplication  @EnableBinding(Processor.class)  public class WordCountApplication {    @StreamListener  @Output("output")  public Flux<?> countWords(@Input("input") Flux<String> words) {  return words.window(ofSeconds(5), ofSeconds(1))  .flatMap(window -> window.groupBy(word -> word)  .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)  .map(count -> new WordCount(group.key(), count))));  } }

22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ [dumb] Pipes and [smart] filters 22 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds

23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 23 http raw-sensor-data averages averages top-n Calculator

24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 24 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub

26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 26 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?

27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS

28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !

29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 28 Averages HDDSHDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other AveragesAverages consumers are competing within a group

31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 30 Average calculator Average calculator http raw-sensor-data

32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 31 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data

33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 32 Average calculator Average calculator source partition 1 partition 2 raw-sensor-data Spring Cloud Stream: Partitioning

34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies: [relatively] easy to deploy … 33 http hdfs spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1

35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 34 http raw-sensor-data averages top-n Calculator Failure detector averagesaverages HDFSHDFSHDFS

36. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 35

37. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 36 Stream definition Spring Boot Apps built with Spring Cloud Stream httpfile = http | file |

38. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell

42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38 cassandra gpfdist http stream1 = http | count-words | file stream2 = jms | cassandra Kafka Data Flow Server DB jms file jms http Message System Platform cassandra count-words

43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39

44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http

45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http LoadBalancer

46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work LoadBalancer

47. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer

54. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties

55. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties app.http.count=2 app.work.count=3 app.hdfs.count=4 app.http.producer.partitionKeyExpression=payload.id

56. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Resource Management 40 http http work work work app.work.spring.cloud.deployer.cloudfoundry.memory=2048

57. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some links … http://cloud.spring.io/spring-cloud-stream http://cloud.spring.io/spring-cloud-dataflow https://github.com/spring-cloud/spring-cloud-stream-samples

58. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ?

Cloud-Native Streaming and Event-Driven Microservices

Estás leyendo una previsualización.

Eche un vistazo a continuación

Cloud-Native Streaming and Event-Driven Microservices

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Cloud-Native Streaming and Event-Driven Microservices (20)

Más de VMware Tanzu (20)

Más reciente (20)