1) The document discusses how Nielsen moved from using Kafka's offset manager to store offsets to using an RDS database for offset storage. This provided more consistent committing of offsets and allowed subscribing to topics with different offset strategies. 2) Key aspects of the new approach included building an offsets map with topic and partition information, getting current offsets from the database, and implementing a "two phase commit" to the database and Kafka in a consistent way. 3) Benefits of the new offset storage approach included faster restart times, improved monitoring, better disaster recovery, and unified configurations.

1. Nielsen Presents: Fun
with Kafka, Spark and
Offset Management
or how we stopped worrying and started to enjoy offset
management

2. Agenda

3. Agenda
• How we used to work
• What we wanted to do and why
• What we ended up doing
• Extra Benefits

4. • Simona
• Big Data Engineer at Nielsen Marketing Cloud
• Data lover
• Concert Goer
• Japan enthusiast
whoami

5. How we used to
work
• Kafka Spark Consumer
• Offsets Strategies
• Kafka Offset Manager
• TEAM A vs TEAM B Flows

6. Our Different Flows
TEAM RON TEAM ITAI
Micro Batch
Length
4-6 minutes 1 hour
Output to
Commit
Kafka
Offsets
Kafka
Offsets, File
names

7. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

8. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

9. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

10. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

11. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

12. Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

13. kafka offset
manager
• Self made
• Offsets stored in kafka
• Everything is against brokers

14. What we wanted to
do and why
• JUST A SIMPLE UPGRADE
• Two main goals to achieve with our
infrastructure
• Some of the problems
we encountered

15. Just a Simple
Upgrade
● New Kafka Consumer API
● New consumer config
● Subscribing to Kafka

16. SOME OF THE PROBLEMS
WE ENCOUNTERED
Committing offsets to Kafka fails
Longer timeouts then!
Spark graceful shutdown
Spark Kafka aSync Commit
……?

17. what we ended up
doing
• RDS Offset Store
• Consistent Committing
• Subscribing to topics (Offsets
Strategies)
• Unified infrastructure
• Timeouts

18. RDS Offset Store
• Table Structure
• Constraints and upserts
• Triggers

19. RDS Offset Store

20. RDS Offset Store

21. RDS Offset Store

22. Subscribing to
Topics

23. Building the
Offsets Map
● What is the offsets Map[TopicPartition,Long]
● Getting the number of partitions
● Constructing the offsets map

24. Getting Number of
Partitions

25. Getting Current
Offsets

26. Getting Current
Offsets

27. Getting Current
Offsets

28. Getting Current
Offsets

29. Getting Current
Offsets

30. Getting Current
Offsets

31. Getting Current
Offsets

32. Getting Current
Offsets
● Each map represents a row
● Reduce rows into the offsets map

33. “Two Phase”
Commit
● What is a “two phase” commit?
● Why do we need it?
● How did we end up implementing it?

34. “Two Phase”
Commit

35. “Two Phase”
Commit

36. “Two Phase”
Commit

37. “Two Phase”
Commit

38. “Two Phase”
Commit

39. “Two Phase”
Commit

40. Extra benefits
• Restart Time
• Monitoring
• Disaster Recovery
• Unified Configurations

41. Committing to
Kafka
● Why?
● Two options of committing to Kafka

42. New Offsets
Strategies
• DataBase
• Earliest
• Latest

43. Working against
bd-commons