1. You Put WHAT in Your Stream?!
Patterns and Practices for Event Design

2. @AdamBellemare | developer.confluent.io
Who am I?
● Staff Technologist at Confluent
● Previously a Data Platform Engineer
● Written a few Books
● Check out the Practical Data Mesh!
Released 2020
AVAILABLE NOW!

3. @AdamBellemare | developer.confluent.io
So you’re at an event-driven architectures talk…

4. @AdamBellemare | developer.confluent.io
“But what did you actually put in your event?”

5. @AdamBellemare | developer.confluent.io
The hand-waving event design technique
Source: NHLGIFs, apparently

6. @AdamBellemare | developer.confluent.io
Okay it didn’t work at all. And my arms are tired.

7. @AdamBellemare | developer.confluent.io
And no, not by going to 7th Normal Form

8. Use Schemas

9. @AdamBellemare | developer.confluent.io
Schema-on-Read Doesn’t Work for Streams
A typical
“Schema-On-Read”
big data file system
Source: https://commons.wikimedia.org/wiki/File:Messy_storage_room_with_boxes.jpg

10. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:

11. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:
True, False

12. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:
True, False, “Yes”, “No”

13. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0

14. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0, null

15. @AdamBellemare | developer.confluent.io
Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0, null
Source: Beaker

16. @AdamBellemare | developer.confluent.io
Per-Consumer Sensitivity to Bad Data
Stream
Stream

17. @AdamBellemare | developer.confluent.io
Bloopers
promotedContent:
True, False,

18. @AdamBellemare | developer.confluent.io
Bloopers
promotedContent:
True, False, 4291, null,

19. @AdamBellemare | developer.confluent.io
Bloopers
promotedContent:
True, False, 4291, null, “null”

20. @AdamBellemare | developer.confluent.io
Bloopers
promotedContent:
True, False, 4291, null, “null”
Source: Uh some kid?

21. @AdamBellemare | developer.confluent.io
Fixing the Bloopers
{
"type" : "record",
"name" : "SampleEvent",
"fields" : [
{ "name" : "isScreenOn" , "type" : "boolean" },
{ "name" : "promotedContent" , "type" : "boolean" }
]
}

22. @AdamBellemare | developer.confluent.io
Schemaless events
Source: Steve’s Disco and Dumpster Inferno Emporium

23. @AdamBellemare | developer.confluent.io
Use a Schema Registry
Confluent Schema Registry
● Supports:
○ Avro
○ Protobuf
○ JSON Schema
● Supports Schema Evolution
● Serialization & Code Generators
● See the Schema Registry 101 Course on developer.confluent.io

24. Factor 1:
Fact or Delta Event Types

25. @AdamBellemare | developer.confluent.io
Fact Events vs. Delta Events
Deltas Facts

26. @AdamBellemare | developer.confluent.io
Fact and Delta Event Types
Cart Fact Event item_added_to_cart
Delta Event
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
{
"cart_id": 3125123,
"item_id": 521,
"quantity": 1
}

27. @AdamBellemare | developer.confluent.io
When do you use Deltas, and when do you use Facts?

28. @AdamBellemare | developer.confluent.io
TIP: Ask yourself:
Are you transferring data?
OR
Are you notifying of an occurrence?

29. @AdamBellemare | developer.confluent.io
A lot of it depends on the boundaries
Internal World
External
World
Service
Boundary
(aka: Bounded
Context)
External
World
External
World

30. @AdamBellemare | developer.confluent.io
The Internal World
Internal World
Service
Boundary
(aka: Bounded
Context)

31. @AdamBellemare | developer.confluent.io
Data on the Inside
Tables
Stream Stream
Event Sourcing
Database-Backed System Event-Sourced System
Private and Purpose-Built for Internal Usage!

32. @AdamBellemare | developer.confluent.io
The External World
Internal World
External
World
External
World
External
World
Service
Boundary
(aka: Bounded
Context)

33. @AdamBellemare | developer.confluent.io
Data on the Outside
Stream
Stream
Stream
Stream
Stream Internal World
External
World
External
World
External
World

34. @AdamBellemare | developer.confluent.io
Events as Data Transfer Objects
Event Stream
Tables
Stream Stream
Database Backed
Event Sourcing
Create External Event
from Internal Model

35. @AdamBellemare | developer.confluent.io
Cart Fact Event (a DTO)
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
item_added_to_cart
{
"cart_id": 3125123,
"item_id": 521,
"quantity": 1
}
discount_code_applied
{
"cart_id": 3125123,
"discount_code": "SAVE-20-2022",
"discount_percent": "20"
}
Facts Model State – Deltas Model Change
Delta
Events

36. @AdamBellemare | developer.confluent.io
Deltas Are Used in Event Sourcing
(Data on the Inside)
Customer Product Quantity
Robert
Pants 1
T-Shirts 1
Hats 15
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read
Apply in order to build state
Current Consumer State
Stream Table

37. @AdamBellemare | developer.confluent.io
Deltas Not Suitable for Building External State
(Data on the Outside)
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read Duplicate
Cart Building
Logic
Independent Consumer
Services
Apply in order to build state
Apply in order
to build state
Stream

38. @AdamBellemare | developer.confluent.io
Use Case: General Alerting of Changes
item_added_to_cart
item_removed_from_cart
discount_code_applied
Consumers

39. @AdamBellemare | developer.confluent.io
What about communicating changes with facts?

40. @AdamBellemare | developer.confluent.io
Inferring Changes from Fact Events
{
"cart_id": 3125123,
"item_map": [ (521,1) ],
"shipping": "standard"
}
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
Cart Fact 1 Cart Fact 2

41. @AdamBellemare | developer.confluent.io
Facts with Before/After Fields
{
"before": {
"cart_id": 3125123,
"item_map": [ (521,1) ],
"shipping": "standard"
},
"after": {
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
}
Cart Fact Event

42. @AdamBellemare | developer.confluent.io
Fact Event Considerations
Event Size

43. @AdamBellemare | developer.confluent.io
Fact Event Considerations
Event Size Update Frequency

44. @AdamBellemare | developer.confluent.io
Fact Event Considerations
Event Size Update Frequency
Intent
?

45. @AdamBellemare | developer.confluent.io
Composite Events?

46. @AdamBellemare | developer.confluent.io
Composite Events: Facts with a Delta?
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "standard"
"reason": "item_added_to_cart"
}
Composite Cart Event

47. @AdamBellemare | developer.confluent.io
Composite Events: Delta with some Facts?
{
"cart_id": 3125123,
"item_id": 923,
"quantity": 3,
"current_cart": {
"item_map": [ (521,1), (923,3) ],
"shipping": "standard"
}
}
item_added_to_cart

48. @AdamBellemare | developer.confluent.io
Facts are good for state transfer and flexibility
Deltas are good for event sourcing
and notifications

49. Factor 2:
How normalized should your events be?

50. @AdamBellemare | developer.confluent.io
Highly Normalized Streams
Item Stream
Brand Stream
Rating Stream
Item
Brand
Rating
Connectors

51. @AdamBellemare | developer.confluent.io
Streaming Joins Can Be Costly
Item Stream
Brand Stream
Rating Stream
Same Joins!
Item
Brand
Rating
Consumers

52. @AdamBellemare | developer.confluent.io
Denormalizing Streams - Everybody’s doing it
Source: The 60s. Maybe the early 70s. That’s like 50 years ago…

53. @AdamBellemare | developer.confluent.io
Events are best served denormalized

54. @AdamBellemare | developer.confluent.io
Option 1: Outbox Table Pattern
Item
Brand
Connector
Outbox
Table
Enriched Item Stream
Transaction
Rating

55. @AdamBellemare | developer.confluent.io
Option 1: Outbox Table Pattern
Item
Brand
Rating
Connector
Outbox
Table
Enriched Item Stream
Transaction
Join
Triggers?

56. @AdamBellemare | developer.confluent.io
Option 1: Outbox Table Pattern
Item
Brand
Rating
Connector
Outbox
Table
Enriched Item Stream
Transaction
Extra Load
Join
Triggers?

57. @AdamBellemare | developer.confluent.io
Option 1: Outbox Table Pattern
Item
Brand
Rating
Connector
Outbox
Table
Enriched Item Stream
Transaction
Code
Extra Load
Join
Triggers?

58. @AdamBellemare | developer.confluent.io
Option 1: Outbox Table Pattern
Item
Brand
Rating
Connector
Outbox
Table
Enriched Item Stream
Transaction
Deletions
Extra Load
Join
Triggers?
Code

59. @AdamBellemare | developer.confluent.io
Option 2: Denormalize the Streams
Item Facts Stream
Brand Facts Stream
Rating Facts Stream
Enriched Item Facts Stream
Join events together
using ksqlDB

60. @AdamBellemare | developer.confluent.io
Option 2: Denormalize the Streams
Normalized
SELECT
I.ID,
I.PRICE,
I.NAME,
B.NAME
FROM ITEM I
INNER JOIN BRAND B
ON I.BRAND_ID = B.ID
EMIT CHANGES;
ksqlDB
Item Facts Stream
Brand Facts Stream
Enriched Item Facts Stream
Denormalized

61. @AdamBellemare | developer.confluent.io
Denormalization Considerations
Event Size

62. @AdamBellemare | developer.confluent.io
Denormalization Considerations
Event Size Join Trigger
Frequency

63. Factor 3:
Single vs. Multiple
Definitions per Stream

64. @AdamBellemare | developer.confluent.io
Use Case 1: Transferring State with Facts
Cart Fact Event
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
Single Type per Stream
Cart Facts
Consumers

65. @AdamBellemare | developer.confluent.io
Use Case: Transferring State with Facts
Cart Facts
Item Facts
Single Type per Stream
Consumers

66. @AdamBellemare | developer.confluent.io
Single Fact Type per Stream
CREATE TABLE Items(
id BIGINT PRIMARY KEY,
price DECIMAL(10,2),
name STRING,
description STRING,
brand_id BIGINT
) WITH (
kafka_topic = 'items',
key_format = 'KAFKA',
value_format = 'AVRO');
Item Facts
ksqlDB

67. @AdamBellemare | developer.confluent.io
Best Practice: Record the Entire State in one Fact
Order
Cart
User
ksqlDB
Order_Cart
Cart
Order_User
User

68. @AdamBellemare | developer.confluent.io
Use Case 2: General Alerting of Changes
item_added_to_cart
item_removed_from_cart
discount_code_applied
Single Type per Stream
Consumers

69. @AdamBellemare | developer.confluent.io
Remove
item
from
cart
Add
item
to
cart
Apply
discount
code
Remove
item
from
cart
Apply
discount
code
Remove
item
from
cart
Cart Events Stream
Use Case 3: Strict Order Processing

70. @AdamBellemare | developer.confluent.io
Multiple Types per Stream
Cart Events Stream
Consumer
item_added_to_cart
item_removed_from_cart
discount_code_applied
Use Case 3: Strict Order Processing

71. @AdamBellemare | developer.confluent.io
Can always split up the stream later
Cart Events Stream
ksqlDB
Application
item_added_to_cart
item_removed_from_cart
discount_code_applied

72. @AdamBellemare | developer.confluent.io
Facts:
Single Definition per Stream
Deltas:
Single or Multiple Definitions per Stream

73. Rapid Fire Best Practices

74. @AdamBellemare | developer.confluent.io
Evolving your Data Model?

75. @AdamBellemare | developer.confluent.io
Versioned Streams for Breaking Changes
Cart Version 2
Cart Version 1
Producer
(Writes both for a period)

76. @AdamBellemare | developer.confluent.io
Versioned Streams for Breaking Changes
Cart Version 2
Producer

77. @AdamBellemare | developer.confluent.io
Deltas
Naming so far?
item_added_to_cart
item_removed_from_cart
discount_code_applied
Facts
Cart
Item
User

78. @AdamBellemare | developer.confluent.io
Streams Across the Business
Inventory
Orders
Shipments
Finance

79. @AdamBellemare | developer.confluent.io
Event Stream Naming
1) <domain>.<event-type>.<version>

80. @AdamBellemare | developer.confluent.io
Event Stream Naming
1) <domain>.<event-type>.<version>
Old Fact Stream Name:
Customers.Advertisement
New Fact Stream Names:
Customers.Advertisement.v2

81. @AdamBellemare | developer.confluent.io
2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name

82. @AdamBellemare | developer.confluent.io
2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name
Orders.Order_Placer_Service.Order.v1
Customers.Ad_Manager.Advertisement.v2
Customers.Ad_Budget_Manager.Ad_Promo_Budget.v2

83. @AdamBellemare | developer.confluent.io
Check out our
Data Mesh eBook!
AVAILABLE NOW!
Stay Tuned for
More Free Courses!