1. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A Kafka Client’s Request
Or, There and Back Again

2. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A Schema
{
"doc": "Accounting for the whereabouts and current
activities of hobbits.",
"fields": [
{
"doc": "Name of the hobbit in question.",
"name": "hobbit_name",
"type": "string"
},
{
"doc": "Current location of the hobbit.",
"name": "location",
"type": "string"
},
{
"doc": "Current status of the hobbit.",
"name": "status",
"type": {
"name": "Status",
"type": "enum",
"symbols": ["EATING", "NAPPING", "SMOKING",
"ADVENTURING", "THIEVING"]
}
}
],
"name": "hobbitUpdate",
"type": "record"
}

3. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A Schema and a Topic
{
"doc": "Accounting for the whereabouts and current
activities of hobbits.",
"fields": [
{
"doc": "Name of the hobbit in question.",
"name": "hobbit_name",
"type": "string"
},
{
"doc": "Current location of the hobbit.",
"name": "location",
"type": "string"
},
{
"doc": "Current status of the hobbit.",
"name": "status",
"type": {
"name": "Status",
"type": "enum",
"symbols": ["EATING", "NAPPING", "SMOKING",
"ADVENTURING", "THIEVING"]
}
}
],
"name": "hobbitUpdate",
"type": "record"
}
Topic hobbit-updates:
● "num.partitions"=6
● "replication.factor"=3
● "min.insync.replicas"=2
● "cleanup.policy"=delete

4. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Producing an Event
key: "Bilbo Baggins"
{
"hobbit_name": "Bilbo Baggins",
"location": "Bag End",
"status": "EATING"
}
producer.send()

5. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
So the data is in Kafka, right?

6. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Let’s go on a journey…

7. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

8. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
0: Within the Producer Client
A. Serializing
○ key.serializer
○ value.serializer

9. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
0: Within the Producer Client
A. Serializing
○ key.serializer
○ value.serializer
B. Partitioning

10. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Partitioning Configurations
● partitioner.class
○ None
■ Hash of key, if present
■ Otherwise, use sticky partition*
○ RoundRobinPartitioner
○ Customize using the Partitioner interface
● partitioner.ignore.keys
● Others to know:
○ partitioner.adaptive.partitioning.enable
○ partitioner.availability.timeout.ms

11. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A. Serializing
○ key.serializer
○ value.serializer
B. Partitioning
C. Batching
0: Within the Producer Client

12. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Batching Configurations
● batch.size
○ Default 16 KB (no batching when set to 0)
○ Batches may not be full
● linger.ms
○ Default 0 ms (no batching when set to 0)
○ Directly affects latency, e.g. linger.ms=10 adds up to 10 ms of latency
● buffer.memory
○ Default ~32 MB
○ Should be > batch.size
○ Chunked into segments of batch.size

13. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Batching Metrics
● batch-size-avg
● records-per-request-avg
● record-size-avg
● buffer-available-bytes
● record-queue-time-avg

14. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A. Serializing
○ key.serializer
○ value.serializer
B. Partitioning
C. Batching
D. [Compressing]
0: Within the Producer Client

15. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
A. Serializing
○ key.serializer
○ value.serializer
B. Partitioning
C. Batching
D. [Compressing]
E. Request
0: Within the Producer Client

16. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Request Configurations
● max.request.size
○ Default ~1 MB
○ Limits number of record batches
● acks
○ Default “all”
○ How many replicas should write the data before sending a response back?
● max.in.flight.requests.per.connection
○ Default 5
○ Limit on unacknowledged requests per broker
● enable.idempotence and transactional.id
● request.timeout.ms
○ Default 30 seconds
○ Time between issuing request and retrying or throwing exception
○ Retrying set handled by delivery.timeout.ms, retries, and retry.backoff.ms

17. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Request Metrics (General)
● request-rate
● requests-in-flight
● request-latency-avg

18. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Produce Request (Bound by max.request.size)
Request Metadata
- transaction ID
- acks
- timeoutMs
Producer Data
Topic Data
dwarf_updates Partition_1
index
Batch
Topic Data
hobbit_updates Partition_0
index
Batch Batch
Partition_4
index
Batch

19. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

20. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Initial request landing zone
● Awaiting processing
● Low-level configurations:
○ socket.receive.buffer.bytes
○ socket.request.max.bytes
1: Socket Receive Buffer

21. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Forms the official produce request
● Adds to the request queue
● Configure and monitor:
○ num.network.threads
○ NetworkProcessorAvgIdlePercent
2: Network threads

22. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Await processing by I/O Threads
● Configure:
○ queued.max.requests
○ queued.max.request.bytes
● Monitor
○ RequestQueueSize
○ RequestQueueTimeMs
3: Request Queue

23. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Finally ready to be processed!
● Data validation
● Configure and monitor:
○ num.io.threads
○ RequestHandlerAvgIdlePercent
4: I/O Threads, aka Request Handler Threads

24. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Underlying commit log
● Segment components
○ .log for data
○ .index for (offset, record)
○ .timeindex
○ .snapshot
5: Page Cache and To Disk

25. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
5. Page Cache/Disk Configuration and Monitoring
● Configure:
○ log.segment.bytes
○ log.flush.interval.ms
○ log.flush.interval.messages
○ [log.]cleanup.policy
● Monitor:
○ LogFlushRateAndTimeMs
○ LocalTimeMs
○ Bonus: monitor page cache performance using system memory

26. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

27. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Hold request until data is has been
replicated as per acks
● Based on hierarchical timing wheel
● Configure:
○ default.replication.factor
○ num.replica.fetchers
○ replica.fetch.wait.max.ms
● Monitor using RemoteTimeMs
6: Mordor Purgatory (but actually)

28. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Await handoff to the network thread
● Unbounded, unconfigurable
● Monitor
○ ResponseQueueSize
○ ResponseQueueTimeMs
7: Response Queue

29. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Places the response on the socket
send buffer
● Last official task of the network thread
8: Network thread hand-off

30. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Final interval on the broker
● Awaiting receipt by producer
● socket.send.buffer.bytes
● Monitor with ResponseSendTimeMs
9: Socket Send Buffer

31. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
● Final summary broker metrics:
○ TotalTimeMs
■ RequestQueueTimeMs
■ LocalTimeMs
■ RemoteTimeMs
■ ResponseSendTimeMs
○ ResponseQueueTimeMs
10: And back again to the Producer

32. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

33. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

34. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Start your own journey…
LinkTree Resources

35. dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/
Questions?