This presentation discusses how Kafka Streams can expose the data from stateful operations, which can enable robust and more streamlined microservices.
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Kick Your Database to the Curb
1. Join the Confluent
Community Slack
Channel
Subscribe to the
Confluent blog
cnfl.io/community-slack cnfl.io/read
Welcome to the Boston Apache Kafka® Meetup!
5:30pm-6:00pm:
Doors Open, Networking, Pizza and
Drinks
6:00pm - 6:45pm:
Bill Bejeck, Confluent
6:45pm-7:00 pm:
Break/Networking
7:00pm - 7:45pm:
Bill Scott, Jacob Zweifel & Srinivas
of Tribal Scale, Cupcakes, Kafka,
and .NET CoreApache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation
with and does not endorse the materials provided at this event.
2. 2
Kick Your Database to the Curb
Using Kafka Streams Interactive Queries to Enable
Powerful MicroServices
3. 3
Brief Introduction
• Worked at Confluent (Streams Team) 2 years
• Apache Kafka Committer
• Author Kafka Streams in Action
Special thanks to @gamussa!
4. 4
Agenda
• What is State
• Kafka Streams Overview
• Describe Interactive Queries
• Live Demo!
7. GroupBy Example
public static void main(String[] args) {
int counter = 0;
int sendInterval = 15;
Map<String, Integer> groupByCounts = new HashMap<>();
try(..consumer = new KafkaConsumer<>(consumerProperties());
..producer = new KafkaProducer<>(producerProperties())){
consumer.subscribe(Arrays.asList(”A”,”B”));
8. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
9. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
15. 15
Stateful Stream Processing
Streams Stateful Operations
• Joins
• Windowing operations
• Aggregation/Reduce
Using any of these operations, Streams creates a state
store
16. 16
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
17. 17
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database
18. 18
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database
23. 23
What’s with the APPLICATION_SERVER_ID
• A single Streams instance doesn’t contain all keys
• Streams will query other instances for store misses
• A single Streams instance can be proxy for all instances
26. 27
Topic Partitions and Streams Tasks
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
Topic with four partitions
Four partitions are converted to 4 tasks so
each streams application is assigned 2
partitions/tasks
27. 28
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B
28. 29
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B
http://hostA:4567?key=FINANCE
29. 30
Example of a Streams RPC
Kafka Streams Application
KAFKA
{ JS }
Demo Time!
30. 31
Embedding the Web Server
KafkaStreams kafkaStreams =
new KafkaStreams(builder.build(), streamsConfig);
InteractiveQueryServer queryServer =
new InteractiveQueryServer(kafkaStreams, hostInfo);
31. 32
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))
32. 33
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))
34. 35
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());
35. 36
Embedding the Web Server
getHostInfo(String storeName, String key) {
StreamsMetadata metadata =
kafkaStreams.metadataForKey(storeName, key, stringSerializer);
return metadata.hostInfo();
}
36. 37
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());
37. 38
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions>
readOnlySessionStore = kafkaStreams.store(
store,
QueryableStoreTypes.sessionStore());
38. 39
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
// Iterate over readOnlySessionStore and
// store results in a list sessionResults
return gson.toJson(sessionResults);
39. 40
Client View Development
<body>
<h2>Kafka Streams Equities Dashboard Application</h2>
<!–- Other div elements left out for clarity -->
<div id="sessionDiv">
<h3 id="sessionHeader">Customer Session Equity Activity Table</h3>
<table id="sessionTable">
<tr>
<th>Customer Id</th>
<th>Average Equity Transaction Spent Per Session</th>
</tr>
</table>
</div>
</body>
40. 41
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>
41. 42
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>
42. 43
Summary
Interactive Queries is a powerful abstraction that
simplifies stateful stream processing
There are still cases for which external database/storage
might be a better
43. 44
Summary
Kafka Streams in Action Examples: https://github.com/bbejeck/kafka-streams-in-
action/blob/master/src/main/java/bbejeck/webserver/InteractiveQueryServer.java
Music example: https://github.com/confluentinc/examples/blob/master/kafka-
streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/
KafkaMusicExample.java
Streaming Movie Ratings: https://github.com/confluentinc/demo-
scene/tree/master/streams-movie-demo
44. 45
Thanks!
Stay in Touch!
• https://slackpass.io/confluentcommunity
• https://www.confluent.io/blog/
• Twitter @bbejeck
• We are hiring! https://www.confluent.io/careers/