Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Microservices Integration Patterns with Kafka

5.351 visualizaciones

Publicado el

Microservice composition or integration is probably the hardest thing in microservices architecture. Unlike conventional centralized ESB based integration, we need to leverage the smart-endpoints and dumb pipes terminology when it comes to integrating microservices.
There two main microservices integration patterns; service orchestration (active integrations) and service choreography (reactive integration). In this talk, we will explore on, Microservice Orchestration, Microservice Choreography, Event Sourcing, CQRS and how Kafka can be leveraged to implement microservices composition

Publicado en: Software
  • Inicia sesión para ver los comentarios

Microservices Integration Patterns with Kafka

  1. 1. Microservices Integration Patterns with Kafka Kasun Indrasiri Director - Integration Architecture, WSO2 Bay Area Apache Kafka Meetup @ Lyft HQ, San Francisco
  2. 2. ● Microservices Integration/composition? ● Microservices Composition Patterns ○ Active/Synchronous Composition ○ Reactive Composition ● Kafka for building Reactive Composition. ● Hybrid of active and reactive composition. Overview
  3. 3. Rise of Cloud Native Applications
  4. 4. Integrating Microservices and Cloud Native Apps Integration is essential to build any real world use case with cloud native apps.
  5. 5. ● APIs: ○ Commands - Actions, synchronous ○ Queries - Request for information, no state change, synchronous ● Events: Facts and notifications, asynchronous ● Streams: A sequence of events/data elements made available over time Microservices Integration/Composition How services can interact over the network?
  6. 6. Active Composition/Orchestration Pattern Synchronous/request-response communication Consumer 1 API Service P API Service Q API Service R API Service S Consumers Consumer 2 Consumer 3 μ Service A Proprietary & Legacy Systems Web API / SaaS API Services/ Edge Services Composite Service/ Integration Services Core Services/ Atomic Services μ Service I μ Service H μ Service J μ Service E μ Service C μ Service D μ Service G μ Service B μ Service F API Management
  7. 7. ● RESTful Services ● gRPC/HTTP2 ● GraphQL ● WebSockets ● Thrift Active Composition/Orchestration Pattern Synchronous/request-response communication
  8. 8. Reactive Composition/Choreography Pattern μ Service A μ Service B μ Service C Event Bus Application Asynchronous event-driven communication Produce Consume Produce Consume Produce Consume Produce Consume
  9. 9. ● Kafka ● AMQP ● JMS Reactive Composition/Choreography Pattern Asynchronous event-driven communication
  10. 10. Kafka in a nutshell Source: “Microservices for Enterprise” https://goo.gl/p3rWF3
  11. 11. ● Consumers from different consumer groups subscribe to the same topic. ● Consumers of a given CG can only consume from one partition. Reactive Composition with Kafka Publisher-subscriber Source: “Microservices for Enterprise” https://goo.gl/p3rWF3
  12. 12. ● Single consumer group and a consumer per each partition. ● Polling and sequential processing of consumer records using offset. Reactive Composition with Kafka Single Consumer/Queuing Source: “Microservices for Enterprise” https://goo.gl/p3rWF3 topic
  13. 13. Delivery Semantics: Exactly-Once/Atomic Broadcast Reactive Composition with Kafka ● You can publish messages and they will be delivered one time exactly by one or more receiving application. ● Producer side ○ Producer writes to the log but fails to get the ack. over network. ● Consumer side ○ Consumer consumes message from the log but fails before updating its offset value. Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f
  14. 14. Delivery Semantics: Exactly-Once/Atomic Broadcast Reactive Composition with Kafka ● Producer side ○ Idempotence : Producers can retry until it is successful ○ Kafka will detect the duplicates and remove them ● Consumer side ○ Store the offsets in the same DB as the derived state and update both in a transaction. ○ Write both state updates and offsets together in a way that is idempotent. Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f
  15. 15. Event Sourcing ● Persist each state changing events of an entity as a sequence of immutable events. ● All such events are stored in an event bus and consumers can derive the state by processing a sequence of events. Shipping
  16. 16. ● Stream Processing in Microservices ○ Services that execute a business logic against a sequence of events/data elements made available over time. ● Kafka Streams : a client library for building stream processing on top of Kafka. Reactive Composition with Kafka Kafka Streams
  17. 17. ● Command Query Responsibility Segregation. ● Split the common data model into query and command models. ● Kafka Stream - Interactive queries. CQRS Command Source: https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Query
  18. 18. ● Kafka does not look at your data but you need consistent across your usage. ○ Avro, ProtoBuf, JSON, XML, ASN.1 ○ Avro preferred schema for Kafka, Kafka schema registry ● Schema in practice ○ Associate an Avro schema with a topic. ○ Declare requirements for data that is produced into the topic ○ Provides the instructions on how to interpret data read from the topic Using Schemas - Avro and Schema Registry
  19. 19. ● A tool for scalably and reliably streaming data between Apache Kafka and other data systems. Reactive Composition with Kafka Kafka Connect Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
  20. 20. Kafka, Kafka Connect and Kafka Streams Source: https://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams/
  21. 21. Active or Reactive? ● Active Composition ○ Ideal for interactive services ○ Service are NOT autonomous ● Reactive Composition ○ Ideal for asynchronous interactions ○ Composition logic is opaque
  22. 22. Hybrid of Active and Reactive Composition Consumer 1 API Service P API Service Q API Service R API Service S Consumers Consumer 2 Consumer 3 μ Service A Event Bus Proprietary & Legacy Systems Web API / SaaS API Services/ Edge Services Composite Service/ Integration Services Core Services/ Atomic Services μ Service I μ Service H μ Service J μ Service E μ Service C μ Service D μ Service G μ Service B API Management
  23. 23. ● Create a queryable materialized view using Kafka Streams or a database with Kafka Connect. Hybrid composition Bridging request-response and event driven messaging Source: https://github.com/confluentinc/kafka-streams-examples/tree/5.0.0-post/src/main/java/io/confluent/examples/streams/microservices
  24. 24. ● A queryable materialized view using Kafka Streams’ interactive queries. Hybrid composition Bridging request-response and event driven messaging
  25. 25. Implementation Technologies ● Java ○ All of the Kafka, Streams and Connect APIs are supported. ○ Confluent microservices example. ● Go, Python, .NET ● Ballerina (ballerina.io) ○ Cloud-native programming languages that makes it easy to write microservices that integrate APIs. https://ballerina.io/learn/by-guide/messaging-with-kafka/ ● Integration Frameworks: Apache Camel, Spring Boot, WSO2 Integrator etc.
  26. 26. THANK YOU Kasun Indrasiri kasun@apache.org

×