Pragmatic Patterns (and Pitfalls) for Event Streaming in Brownfield Environments with Anna McDonald

Anna McDonald: Twitter: @jbfletch_ <- fully committed to the underscore
GitHub: jbfletch
@jbfletch_

ARE YOU IN THE RIGHT TALK?
Does your environment look like...
Super Fancy Half Fancy help.
2
@jbfletch_

@jbfletch_
Why Should I Event Stream at All?

@jbfletch_
● Knowing quickly is actionable

@jbfletch_
● Integration requirements

@jbfletch_
● Integration requirements
● User expectations

@jbfletch_
Potential Problems to Tackle

@jbfletch_
Problem 1. How do I get my data?

@jbfletch_
● Kafka Producer

@jbfletch_
Kafka Producer
● Throwing events directly
● Custom stateful events
from flat files (EBCDIC)
● Available in Java, .Net,
Python, Scala, Kotlin, JS and
many more! No native
COBOL..YET!

@jbfletch_
● Kafka Producer
● Connect

@jbfletch_
Connect
@C0urante
@gharris1727

@jbfletch_
Connect
● Stateless connection for
external systems to Kafka
● Low code with SMT
functionality
● Rich ecosystem of
connectors

@jbfletch_
● Kafka Producer
● Connect
● Change Data Capture

MySQL
inserts: op_type: I,
after struct
Debezium
15
@jbfletch_
Change Data Capture (CDC)

MySQL
after struct
updates: op_type: U,
before,after struct
Debezium
16
@jbfletch_

MySQL
after struct
updates: op_type: U,
before,after struct
deletes: op_type: D,
before struct
Plus optional tombstone
message
Debezium
17
@jbfletch_

@jbfletch_
Problem 2. Moving to Async from
Existing Synchronous Workflows

@jbfletch_
● How do I delay my data by an
hour?
● Is it bad to have 14 retry
topics?
RED FLAGS!
Problem 2. Moving to Async from
Existing Synchronous Workflows

@jbfletch_
Embrace Completion Criteria

@jbfletch_
1. Define the
information you need
for completeness.

@jbfletch_
● Order Placed event
needs header and
items

@jbfletch_
needs header and
items
● Order Filled event
needs header, items,
and stock details

@jbfletch_
needs header and
items
● Order Filled event
and stock details
● Order Shipped event
stock, address and
shipping details

@jbfletch_
1. Define the
for completeness.
2. Enumerate and map
those sources.

@jbfletch_
needs header and items

@jbfletch_
● Order Filled event needs
header, items, and stock
details
Mainframe
+

@jbfletch_
● Order Filled event needs
header, items, and stock
details
● Order Shipped event
stock, address and
shipping details
Mainframe
+
+

@jbfletch_
1. Define the
for completeness.
2. Enumerate and map
those sources.
3. Employ a known best
practice to stop
using time and start
using completion.

@jbfletch_
Example 1.

@jbfletch_
Change Data Capture (CDC)...Multiple Tables Uh Oh..
What do I do when my
commit is across multiple
tables..how do I handle
transactions?

@jbfletch_
Outbox Pattern

@jbfletch_
Derivative Events

Becoming an event profiler
Trigger the event
in the source
system
Review the CDC
Messages
generated during
the event
Find the event
fingerprint that
signifies
completeness
WANTED
34
@jbfletch_

Trigger the event
in the source
system
Review the CDC
Messages
generated during
the event
Find the event
fingerprint that
signifies
completeness
WANTED
35
@jbfletch_

Trigger the event
in the source
system
Review the CDC
Messages
generated during
the event
Find the event
fingerprint that
signifies
completeness
WANTED
36
@jbfletch_

Complex Events
Multiple Tables Multiple Sources
MySQL Oracle
What constitutes event completeness?
37
@jbfletch_

Order and Items example
Order # 42
Total # Items 3
38
@jbfletch_

Plan of attack
40
We need both of these things to be true before we
fire the event:
Order # 42
Total # Items 3
Order Event Aggregate
@jbfletch_

How do we wait for all items to arrive?
41
@jbfletch_

What is the minimum information we need to know to
be able to determine event completeness for items?
42
@jbfletch_
Total Number
of items per
order

Basic Central Event Service Setup
Filter using the event profile: op_type = “I” to create:
KTable<OrderId, Order> orderTableKeyOrderId <- Orders that
match our order created event profile
KStream<ItemId,Items> itemsKeyedByItemIdStream <- Items
that match our order created event profile
KTable<OrderId,NumberOfItems> totalNumberofItemsTable <-
Number of items in each order
44
1
2
3

groupBy + aggregate + join + filter
KTable<OrderId, ArrayList<Items>> preItemsTable =
itemsKeyedByItemIdStream(ItemId,Items)
.groupBy(ORDER_ID) ← OrderId: 42
.aggregate(ArrayList::new, add(Items), return null for TS)
.join(totalNumberofItemsTable) <- itemCount: 3
Optional join allows for the propagation of total order
items to each item
45
1
2
3
@jbfletch_

itemsKeyedByItemIdStream<ItemId,Items>
items to each item
46
1
2
3
@jbfletch_

itemsKeyedByItemIdStream(ItemId,Items)
items to each item
47
1
2
3
@jbfletch_

KTable<OrderId, ArrayList<Items>> fullItemsTable =
preItemsTable
.filter((k,v)-> v.size() ==
v.get(0).get(“itemCount”).asInt()) <- This
filter will block until all 3 item lines are
in the array
48
4
@jbfletch_

Part Deux
49
How can we be sure the order message has arrived?
Order # 42
Total # Items 3
Order Event Aggregate
@jbfletch_

Thou shall not pass!!! Using inner joins
50
KTable fullOrderTable =
orderTableKeyOrderId
.join(fullItemsTable, <- only
contains orders with all items
arrived (orderNode, itemNodes) -> {
construct and return order
placed event aggregate
})
@jbfletch_

@jbfletch_
Problem 3. What if my completion
criteria is never met?

@jbfletch_
Problem 3. What if my completion
criteria is never met?
Absence of
an Event!

@jbfletch_
Absence of an Event Detection with Kafka Streams
Time Exceeded?
Order Header Delay
Event
Order Items Delay
Event
The Processor API (PAPI)
can be used to evaluate
the time an entry has
existed in a state store, if
it’s over the tolerance you
set, an event can be
emitted.

@jbfletch_
Problem 4. Keeping track of complex
workflows

@jbfletch_
Routing Slip!
routingHeader:
● name
● currentStep
● slipId
routingSteps
● stepNumber
● Specific Parameters
● Error routing

Pragmatic Patterns (and Pitfalls) for Event Streaming in Brownfield Environments with Anna McDonald

Recomendados

Recomendados

Más contenido relacionado

Similar a Pragmatic Patterns (and Pitfalls) for Event Streaming in Brownfield Environments with Anna McDonald

Similar a Pragmatic Patterns (and Pitfalls) for Event Streaming in Brownfield Environments with Anna McDonald (20)

Más de HostedbyConfluent

Más de HostedbyConfluent (20)

Último

Último (20)

Pragmatic Patterns (and Pitfalls) for Event Streaming in Brownfield Environments with Anna McDonald