1.
Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
Tech Deep Dive
Message Redelivery:
An Unexpected
Journey
David Kjerrumgaard
Developer Advocate • StreamNative

2.
• Apache Pulsar Committer & author of
Pulsar In Action
• Former Principal Software Engineer on
Splunk’s Pulsar-as-a-Service platform
• Over 15 years experience in Big
Data/Streaming
David Kjerrumgaard
Developer Advocate
StreamNative

3.
Pulsar Client & Broker

4.
streamnative.io
10,000 Foot Level
● Pulsar Clients and Brokers interact with
one another by exchanging Command
Messages back-and-forth.
● Brokers are continually listening for
incoming commands on a known port.
● Clients create a connection to the Broker
on this port and start issuing commands.

5.
streamnative.io
Command Categories
● These commands can be broken down into different categories
based on how they are used.

6.
streamnative.io
Client Commands
● These commands are used by the
Pulsar Client & Broker to establish
communication, authenticate, and
perform topic lookups.
● Commands are used to transition the
client through various states, e.g.,
disconnected to connected.

7.
Client Creation
● When the build() method is called, under the covers
the CONNECT command is sent to the address
specified by the serviceUrl property.
● If the client is authenticated, then the Pulsar Broker
will respond with a CONNECTED command.

8.
streamnative.io
Client Creation Flow
● The builder dispatches a
CONNECT command
(including credentials) to
establish a connection.
● The Broker authenticates
the client and sends a
CONNECTED command to
indicate success.

9.
Message
Publication

10.
streamnative.io
Producer Commands
● These commands are used to move
messages from the Producer to the
Broker.
● Commands are used to perform
message handshaking between the
Producer and the Broker, e.g.,
SEND/SEND_RECEIPT

11.
Producer Creation
● When the create() method is called, under the covers a
PRODUCER command is sent to the Broker.
● If the producer is authorized and has a compatible
schema, then the Pulsar Broker will respond with a
PRODUCER_SUCCESS command.

12.
Producer Creation – Configuration
ProducerBuilder Method Description
enableBatching (boolean enable) Control whether automatic batching of
messages is enabled for the producer.
batchingMaxBytes (int max) Set the maximum number of bytes
permitted in a batch
batchingMaxMessages (int max) Set the maximum number of messages
permitted in a batch.
batchingMaxPublishDelay (long delay,
TimeUnit timeUnit)
Set the time period within which the
messages sent will be batched
maxPendingMessages (int max) Set the max size of the queue holding the
messages pending to receive an
acknowledgment from the broker.
sendTimeout(int timeout, TimeUnit
unit)
If a message is not acknowledged by the
server before the sendTimeout expires, an
error will be reported.

13.
streamnative.io
Producer Creation Flow
● The PRODUCER command
includes additional info such
as the Schema, etc.
● The Broker creates a
Producer object on the
Broker side and sends a
PRODUCER_SUCCESS
command if successful.

14.
Broker-Side Producer Object
● An org.apache.pulsar.broker.service.Producer
object is created on the broker-side that retains a
reference to the corresponding Topic object.
● The Topic object is responsible for writing all the
received messages to the Managed Ledger.

15.
Client-Side Producer Object
● A Producer object is created on the client-side that has a
BatchMessageContainer object that is used to retain
messages until the batch is considered full.
● Messages published via the send()/sendAsync()
methods are stored in the BatchMessageContainer.
● When the batch is full, messages are removed from the
BatchMessageContainer and moved to a
pendingMessages queue to await acknowledgement.

16.
Message Publication
● The producing application
can call the send()
method multiple times.
● Each of these messages
are added to the
BatchMessageContainer
until a batch send is
triggered due to size or
time.

17.
Message Publication – cont.
● Once the batch is considered
“full”, the raw messages are
prepared for transmission to the
Broker.
● Each message is annotated with
metadata such as the
producer's name, sequenceID,
and creation time.

18.
Message Publication – cont.
● The wrapped messages are added to the pendingMessage
queue and sent to the Broker via a SEND command.

19.
Message Publication – Success.
● Once the message is persisted to BookKeeper the Broker
notifies the Producer via a SEND_RECEIPT command and the
message is removed from the pendingMessage queue.

20.
Publication Redelivery Scenario #1
● If the message cannot be persisted, the Broker notifies the
Producer via a SEND_ERROR command and the message will
be resent from the pendingMessage queue.

21.
Publication Redelivery Scenario #1 – cont
● If the PRODUCER receives a SEND_ERROR message, then it
will resend all the messages in the pendingMessages queue.

22.
Publication Redelivery Scenario #2
● If there is an issue during the transmission of the message,
then it remains in the pendingMessages queue and will be
resent will the Producer reconnects.

23.
Publication Redelivery Scenario #2
● After the Producer
reconnects, all the
messages in the
pendingMessages
queue are re-sent
automatically.

24.
Message
Consumption

25.
streamnative.io
Consumer Commands
● These commands are used to
move messages from the
Broker to the Consumer.
● Commands are used to
perform message
handshaking between the
Consumer and the Broker,
e.g., MESSAGE/ACK

26.
Consumer Creation
● When the subscribe() method is called, under the
covers a SUBSCRIBE command is sent to the Broker.
● If the Consumer is authorized and has specified a
compatible schema, then the Pulsar Broker will respond
with a SUCCESS command.

27.
Consumer Creation – Configuration
ConsumerBuilder Method Description
acknowledgmentGroupTime (long delay,
TimeUnit unit)
Group the consumer acknowledgments for the
specified time.
ackTimeout(long ackTimeout,
TimeUnit timeUnit)
Set the timeout for unacked messages
ackTimeoutTickTime (long tickTime,
TimeUnit timeUnit)
Define the granularity of the ack-timeout
redelivery.
ackTimeoutRedeliveryBackoff (Redeli
veryBackoff backoff)
Allows you to specify a custom redelivery
strategy for message that have exceeded the
ack timeout
negativeAckRedeliveryDelay (long
delay, TimeUnit timeUnit)
Set the delay to wait before re-delivering
messages that have failed to be process.
negativeAckRedeliveryBackoff (Redel
iveryBackoff backoff)
Allows you to specify a custom redelivery
strategy for negatively acknowledged messages

28.
streamnative.io
Consumer Creation Flow
● The SUBSCRIBE command
includes additional info such
as the Schema, etc.
● The Broker creates a
Consumer object on the
Broker side and sends a
SUCCESS command if
successful.

29.
Broker-Side Consumer Object

30.
Client-Side Consumer Object

31.
The Message
Delivery
Process

32.
Message Flow
● A pull-based model controlled by the FLOW command.

33.
Message Flow - Backend
● The messages are chosen by the Subscription and read from
BookKeeper.

34.
Message Flow – cont.
● The messages are added to the pendingAcks HashMap
before being sent back via individual MESSAGE commands.

35.
Message Consumption
● When you consume messages using the receive() /
receiveAsync() methods, the messages are taken from the
incomingMessages queue and given to the application.

36.
Automated Message Flow
● The client-side Consumer periodically requests new
messages from the broker-side Consumer (pull-based)
● This process is triggered automatically when the
client-side Consumer’s incomingMessages queue drops
below 50% of its capacity.
● When this condition is detected a FLOW command is
dispatched to the broker-side Consumer to request
another batch of messages.

37.
Post-Processing Options
● After the message is processed, it can either be
acknowledged, negatively acknowledged, or sent to the
retry letter topic to be re-delivered after a specified length
of time.

38.
The Message
Acknowledgement
Process

39.
Message Acknowledgement
● After a message has been successfully consumed, it must
be acknowledged to prevent redelivery/reprocessing.
● The behavior of the Acknowledgment process can be
controlled by the following settings in the ConsumerBuilder
• acknowledgmentGroupTime(long delay, TimeUnit unit)
• ackTimeout(long ackTimeout, TimeUnit timeUnit)
• ackTimeoutRedeliveryBackoff(RedeliveryBackoff ackBackoff)
• ackTimeoutTickTime(long tickTime, TimeUnit timeUnit)

40.
Message Acknowledgement
● If the acknowledgmentGroupTime property is set, then the acks will
be grouped together for the specified interval before being sent
together, otherwise an ack is sent immediately.

41.
Message Acknowledgement - Broker
● On the Broker side, the ack is removed from the pendingAcks list, and the
subscription updates the cursor position. An ACK_RESPONSE command is
sent back to the client to complete the handshake.

42.
Redelivery of
Messages Due to
Ack Timeout

43.
Waiting
● On the client-side, the UnackedMessageTracker tracks the
receipt of the ACK commands that were sent to the Broker to
ensure that the subscription cursor is positioned correctly.
● Any ACKs that do not receive a corresponding ACK_RESPONSE
command must be resent to complete the handshake. Missing
ACK_RESPONSEs indicate the possibility of a network outage, etc.
● How long the client decided to wait for these ACK_RESPONSEs is
referred to as the “Ack Timeout”

44.
The UnackedMessageTracker
● Messages are added to the UnackedMessageTracker as
their corresponding ACKs are sent to the Broker.

45.
The UnackedMessageTracker
● To ensure that we only redeliver messages that have not
been ACKed by the Broker, we remove message as their
corresponding ACK_RESPONSEs are received from the
Broker.

46.
The UnackedMessageTracker
● There are 3 variations of the UnackedMessageTracker, and
the one that is used is determined by the way that the
consumer was configured.

47.
Ack Timeout Configuration
● If the ackTimeout property is set to zero, then ACK tracking
is disabled, and we have AT_MOST_ONCE processing
semantics.
● Otherwise, we use one of the two timer-based variations
that waits a period for the ACK_RESPONSEs before
resending the ACKs to the Broker.
● The only difference between the two is how the ack timeout
interval is calculated.

48.
Timer-Based UnackedMessgeTrackers
● Internally, both timer-based UnackedMessageTracker versions
use a Timer task to trigger the redelivery of the ACKs.
● The base UnackedMessageTracker operates on a fixed interval
equal to either the ackTimeoutTickTime property (if set) or else
the ackTimeout property.
● The UnackedMessageRedeliveryTracker operates on a variable
interval based upon the value of the
ackTimeoutRedeliveryBackoff property.

49.
Timer-Based UnackedMessgeTrackers
● Here is an example of an UnackedMessageRedeliveryTracker
configured to use an Exponential backoff strategy.

50.
Ack Timeout
● Once the ack timeout interval has elapsed, the consumer
sends a REDELIVER_UNACKNOWLEDGED_MESSAGES
command to the Broker.

51.
Ack Timeout
● The REDELIVER_UNACKNOWLEDGED_MESSAGES triggers
the Broker-side consumer to resend ALL the messages in
PendingAcks.

52.
Redelivery of
Negatively Acked
Messages

53.
The NegativeAcksTracker
● Messages are added to the NegativeAcksTracker when
the application calls consumer.negativeAck(msg)

54.
Negative Acks Configuration
● There is only one variation of the
NegativeAcksTracker, and it uses a Timer task to
trigger the redelivery of the negatively acked
messages.
● The Timer operates on a fixed interval if the
negativeAckRedeliveryDelay property is set.
Otherwise, it uses the strategy specified by the
negativeAckRedeliveryBackoff property.

55.
Timeout
● Once the redelivery delay has elapsed, the consumer
sends a REDELIVER_UNACKNOWLEDGED_MESSAGES
command to the Broker.

56.
Redelivery of
Delayed
Messages

57.
Delayed Message Delivery
● If the enableRetry property is set to true, the consumer
can request that a message be re-delivered after a specified
delay, e.g., consumer.reconsumeLater(msg, 3,
TimeUnit.SECONDS)
● Under the covers, this process utilizes the
deliverAfter(delayTime, unit) method of the
Producer class.

58.
Delayed Message Delivery
● When you first call the reconsumeLater method on the
consumer, a Producer object is initialized using the following
code block.

59.
Delayed Message Delivery
● Next, we check to see how many times we have already
tried to redeliver this message by examining the
SYSTEM_PROPERTY_RECONSUMETIMES property in the
Message’s properties.
● If that value exceeds the maxRedeliverCount setting of the
DLQ policy, then the message is routed to the DLQ.
● Otherwise, it is scheduled for delayed deliver via a call to
producer.newMessage().deliverAfter(delayTime,
unit);

60.
streamnative.io
Key Takeaways
➔ Message redelivery can occur
when Producing or Consuming
messages in Pulsar
➔ When producing, messages get
re-delivered if the Broker doesn’t
acknowledge receipt of the
message in a timely manner.
➔ When consuming, messages get
re-delivered under 3 different
circumstances. Ack timeout,
negative ack, and delay

61.
David Kjerrumgaard
Thank you!
david@streamnative.io
@DavidKjerrumga1
Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022