SpringOne 2021: Session Title: Reactive Applications with Apache Pulsar and Spring Boot Speaker: Lari Hotari, Senior Software Engineer at DataStax, Inc

1. Reactive Applications with
Apache Pulsar and Spring Boot
Lari Hotari @lhotari
Senior Software Engineer, DataStax, Inc
Apache Pulsar committer
Presentation at SpringOne 2021
September 2, 2021

2. Who
2
Lari Hotari @lhotari
Software Engineer @DataStax, working on Luna Streaming powered by Apache Pulsar
Open-source contributor, Apache Pulsar committer
Journey with Spring:
○ Spring Framework user since 0.9 (2003)
○ Early Spring Framework contributor (2007, WebSphere DataSource support
for Spring Transaction Management)
○ Grails contributor (2009-2020)
Member of Groovy and Grails team (part of the Spring team) at Pivotal (2014-2015)
Grails team released Grails 3.0 built on Spring Boot in March 2015
○ Spring Boot contributor (heapdump actuator)
○ Project Reactor contributor (dns reverse lookup optimization & few others
in reactor-netty, 1 minor in reactor-core)
2004
2002
2005
2003

3. 3
Pulsar & Reactive
Messaging
01
Sample use case &
Live coding
02
Scaling performance
of message
processing
03
Agenda

4. 4
Pulsar & Reactive
Messaging
01
Sample use case &
Live coding
02
Scaling performance
of message
processing
03
Agenda

5. Apache Pulsar - Favourite properties
5
Many options for
scaling - not just about
partitions
02
Streaming, pub-sub
and queuing come
together
01
Cloud Native
Architecture
Layered storage
architecture
Stateless Brokers
First class support
for Kubernetes
03

6. Reactive Message handling - why?
6
Fault tolerance with
fallbacks, retries and
timeouts
Compatibility with
Reactive Streams to
achieve reactive from
end-to-end
Performance by
parallelism and
pipelining in
processing
Resilience with flow
control (backpressure)
Efficiency by optimal
resource usage
Simplification by
data orientation &
functional approach

7. Reactive Pulsar Adapter library
● https://github.com/lhotari/reactive-pulsar , License: ASL 2.0
● Wraps the Apache Pulsar Java Client async API with a simple and intuitive reactive API
● Goes beyond a wrapper
● Back-pressure support for provided interfaces
● Pulsar client resource lifecycle management
● Message producer caching
● In-order parallel processing at the application level
● Spring Boot starter for auto configuring a Pulsar Java client and a ReactivePulsarClient
7

8. Reactive Messaging Application building blocks
provided by the Reactive Pulsar Adapter library
8
Message Sender
Message Consumer
Message Listener Container
Message Reader
● Sender - for sending messages with a specific
configuration to a specific destination topic.
● Consumer - for guaranteed message delivery and handling.
The broker redelivers unacknowledged messages. Used for
both “always on” use cases and batch processing or
short-time message consuming or polling use cases.
● Reader - The application decides the position where to start
reading messages. Position can be earliest, latest, or an
absolute or time based position. Suitable also for
short-time message polling.
● Message Listener Container - integrates a consumer with
the Spring application lifecycle.
*) The abstractions of the Reactive Pulsar Adapter library are slightly different from the abstractions in Pulsar Java Client. The sender
is one example of a difference. The lifecycles of the abstractions are completely different. The ReactiveMessageSender,
ReactiveMessageReader and ReactiveMessageConsumer instances themselves are stateless.
“Nothing happens until you subscribe” - the key difference in the Reactive programming model.

9. Basics of Reactive APIs
9
- “Nothing happens until you subscribe”
- Assembly-time vs Runtime
- API calls return a reactive publisher type
- Flux<T> or Mono<T> when using Project Reactor
- Mono<Void> for calls that don’t return data

10. Reactive Pulsar Adapter
public interface ReactivePulsarClient {
<T> ReactiveMessageSenderBuilder<T> messageSender(Schema<T> schema);
<T> ReactiveMessageReaderBuilder<T> messageReader(Schema<T> schema);
<T> ReactiveMessageConsumerBuilder<T> messageConsumer(Schema<T> schema);
}
10
public interface ReactiveMessageReader<T> {
Mono<Message<T>> readMessage();
Flux<Message<T>> readMessages();
}
public interface ReactiveMessageSender<T> {
Mono<MessageId> sendMessage(Mono<MessageSpec<T>> messageSpec);
Flux<MessageId> sendMessages(Flux<MessageSpec<T>> messageSpecs);
}
public interface ReactiveMessageConsumer<T> {
<R> Mono<R> consumeMessage(
Function<Mono<Message<T>>, Mono<MessageResult<R>>> messageHandler);
<R> Flux<R> consumeMessages(
Function<Flux<Message<T>>, Flux<MessageResult<R>>> messageHandler);
}

11. Sample use cases - simplified IoT backend
11
Telemetry ingestion
from IoT devices
Telemetry processing
Streaming of telemetry values
to other applications
Sending alerts to
external application - webhook
interface
https://github.com/lhotari/reactive-pulsar-showcase for complete sample (work in progress)

12. 12
Pulsar & Reactive
Messaging
01
Sample use case &
Live coding
02
Scaling performance
of message
processing
03
Agenda

13. Our goal for live coding
13

14. 14
Pulsar & Reactive
Messaging
01
Sample use case &
Live coding
02
Scaling performance
of message
processing
03
Agenda

15. All significant improvements in
system scalability come from
parallelism.
15

16. Pipelining
16
“This idea is an extension of the idea of
parallelism. It is essentially handling the
activities involved in instruction execution
as an assembly line. As soon as the first
activity of an instruction is done you move
it to the second activity and start the first
activity of a new instruction. This results in
executing more instructions per unit time
compared to waiting for all activities of the
first instruction to complete before
starting the second instruction.”
https://www.d.umn.edu/~gshute/arch/great-ideas.html

17. Parallel Pipelining with Project Reactor:
Without ordering requirements
17
● When there are no ordering requirements, parallelism/concurrency can be achieved in two ways:
Using .parallel operator
(parallelism, for computations)
Flux
.range(1, 10)
.parallel(10)
.runOn(Schedulers.parallel())
.concatMap(i -> {
// CPU bound
…
// that returns a publisher
}
)
.sequential()
.subscribe()
Using .flatMap + .subscribeOn for inner publisher
(concurrency, for I/O tasks)
Flux
.range(1, 10)
.flatMap(
i -> {
// IO bound
…
// that returns a publisher
}.subscribeOn(Schedulers.parallel()),
10
)
.subscribe()

18. In-order parallel processing strategies
● System level: Multiple topic partitions
● Message is mapped to a specific partition by hashing the key
partition = hash(message key) % number_of_partitions
● Application level: Message consumer parallelism
● Micro-batching
● Messages are consumed in batches which is limited by time and number of entries.
● The processing can sort and group the entries for parallel processing.
● Stream splitting / routing / grouping
● Message is mapped to a specific processing group based on the hash of the message key
processing group = hash(message key) % number_of_processing_groups
● This is well suited for splitting the stream with Project Reactor’s .groupBy
● Benefit over micro-batching: no extra latency caused by batching
18

19. Parallel Pipelining with Project Reactor:
In-order processing requirements
19
When there is a requirement of in-order processing by key:
● .groupBy operator splits the stream by a key. The total
number of keys should be limited and relatively small
(hundreds).
● The concurrency level parameter for flatMap should be
set to the total number of groups to prevent the stream
processing from stalling (explained in Flux.groupBy
javadoc). In addition, the “prefetch” parameter of
.groupBy should be set to a value >= total number of
groups.
Flux
.range(1, 10)
// simulation of key: 0 (even) or 1 (odd)
.groupBy(i -> i % 2, Math.max(10, 32))
.flatMap(
groupedFlux ->
groupedFlux
.publishOn(Schedulers.parallel())
.concatMap(x -> {
// IO bound
…
// that returns a publisher
})
10,
1
)
.subscribe()

20. Parallel processing with ReactiveMessageConsumer
20
● An enabler is the the special signature of the
consumeMessages method on the
ReactiveMessageConsumer interface.
● The method accepts a function that takes
Flux<Message<T>> input and produces
Flux<MessageResult<R>>> output.
● MessageResult combines
acknowledgement and the result. The
implementation consumes the
Flux<MessageResult<R>>> , handles the
acknowledgements, and returns a Flux<R> .
● This enables guaranteed message delivery
combined with stream transformations.
public interface MessageResult<T> {
static <T> MessageResult<T> acknowledge(MessageId messageId, T value) {}
static <T> MessageResult<T> negativeAcknowledge(MessageId messageId, T
value) {}
static MessageResult<Void> acknowledge(MessageId messageId) {}
static MessageResult<Void> negativeAcknowledge(MessageId messageId) {}
static <V> MessageResult<Message<V>> acknowledgeAndReturn(Message<V>
message) {}
boolean isAcknowledgeMessage();
MessageId getMessageId();
T getValue();
}
public interface ReactiveMessageConsumer<T> {
<R> Flux<R> consumeMessages(
Function<Flux<Message<T>>, Flux<MessageResult<R>>>
messageHandler);
}

21. In-order parallel processing with Reactive Pulsar Adapter:
Implementation details
21
private static Integer resolveGroupKey(Message<?> message, int concurrency) {
byte[] keyBytes = null;
if (message.hasOrderingKey()) {
keyBytes = message.getOrderingKey();
} else if (message.hasKey()) {
keyBytes = message.getKeyBytes();
}
if (keyBytes == null || keyBytes.length == 0) {
// derive from the message id
keyBytes = message.getMessageId().toByteArray();
}
int keyHash = Murmur3_32Hash.getInstance().makeHash(keyBytes);
return keyHash % concurrency;
}
messageFlux
.groupBy(message -> resolveGroupKey(message, concurrency), Math.max(concurrency, 32))
.flatMap(
groupedFlux ->
groupedFlux
.publishOn(Schedulers.parallel())
.concatMap(message -> handleMessage(message)),
concurrency
);
actual implementation has been refactored:
https://github.com/lhotari/reactive-pulsar/blob/master/reactive-pulsar-adapter/src/main/java/com/github/lhotari/reactive/pulsar/internal/adapter/InKeyOrderMessageProcessors.java

22. References
22
Apache Pulsar: https://pulsar.apache.org/
Spring Reactive: https://spring.io/reactive
Reactive Pulsar adapter: https://github.com/lhotari/reactive-pulsar
Status: experimental version 0.0.7 available for use, API is subject to change until 1.0 is
released.
Reactive Pulsar showcase application: https://github.com/lhotari/reactive-pulsar-showcase
Status: work in progress, demonstrates the usage of the Reactive Pulsar Adapter.
For usage questions, please use apache-pulsar and reactive-pulsar tags on Stackoverflow.

23. Thank you !
23
We are hiring: https://www.datastax.com/company/careers