1. Versioned State Stores in Kafka Streams
Victoria Xia - Confluent
1

2. ● Primer + Motivation
● Intro to Versioned Stores
● Examples
● Getting Started
● Things to Know
● Implementation (in Brief)
● What’s Next
Outline
2

3. Primer: State Stores in Kafka Streams
3

4. Primer: State Stores in Kafka Streams
4

5. Primer: State Stores in Kafka Streams
5

6. Primer: State Stores in Kafka Streams
6

7. Introducing: Versioned Stores
7
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)

8. Introducing: Versioned Stores
8
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)

9. Introducing: Versioned Stores
9
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)
● History retention: how far back in time reads may occur

10. Introducing: Versioned Stores
10
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)
● History retention: how far back in time reads may occur

11. Introducing: Versioned Stores
11
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)
● History retention: how far back in time reads may occur

12. Introducing: Versioned Stores
12
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)
● History retention: how far back in time reads may occur

13. Introducing: Versioned Stores
13
● Store multiple record versions per key
● Timestamped lookups: get(key, asOfTimestamp)
● History retention: how far back in time reads may occur

14. Versioned Stores in Action: Stream-Table Join
14

15. Versioned Stores in Action: Table Aggregation
15

16. Versioned Stores in Action: Table-Table Join
16

17. Getting Started
17
● Available starting in AK 3.5
● Purely opt-in

18. Getting Started: DSL
18
streamsBuilder
.table(
topicName,
Materialized.as(
Stores.persistentVersionedKeyValueStore(
STORE_NAME,
Duration.ofMillis(HISTORY_RETENTION)
)
)
);

19. So What’s a “Versioned Table,” Anyway?
19

20. So What’s a “Versioned Table,” Anyway?
20

21. So What’s a “Versioned Table,” Anyway?
21

22. So What’s a “Versioned Table,” Anyway?
22

23. So What’s a “Versioned Table,” Anyway?
23

24. So What’s a “Versioned Table,” Anyway?
24

25. So What’s a “Versioned Table,” Anyway?
25

26. So What’s a “Versioned Table,” Anyway?
26

27. So What’s a “Versioned Table,” Anyway?
27

28. So What’s a “Versioned Table,” Anyway?
28

29. So What’s a “Versioned Table,” Anyway?
29
A table is versioned if:
● Materialized with a versioned store
● No intermediate (un-versioned) materialization or stateful
transformation

30. Getting Started: Processor API
30
streamsBuilder
.addStateStore(
Stores.versionedKeyValueStoreBuilder(
Stores.persistentVersionedKeyValueStore(
STORE_NAME,
Duration.ofMillis(HISTORY_RETENTION)),
Serdes.Integer(),
Serdes.String()
)
);

31. Things to Know
31
● Interactive queries not yet supported
● No in-memory implementation yet
● Performance implications

32. Versioned Store Implementation
32

33. Versioned Store Implementation
33

34. Versioned Store Implementation
34
—> adjust segment interval to tune performance

35. Future Work
35
● Interactive queries
● In-memory implementation
● Store-specific metrics
● Separate “grace period” from history retention
● Optionally enable versioned stores by default
● Additional join semantics improvements (e.g., KIP-923)

36. Takeaways
36
● Timestamped lookups: get(key, asOfTimestamp)
● DSL processors handle out-of-order data using
timestamp-based semantics
● Opt in starting with AK 3.5
Learn More
● KIP-889: Versioned State Stores
● KIP-914: DSL Processor Semantics for Versioned Stores

37. 37