Apache Kafka is a distributed streaming platform that can be used to build real-time data pipelines. It publishes and subscribes to streams of records in a fault-tolerant and durable way, and helps process streams of records as they occur. Key characteristics of Kafka include high throughput ingestion, fault-tolerant storage, high availability, scalability, and support for concurrent processing and ordering guarantees. It provides functionality similar to a messaging system but with a unique design as a distributed, partitioned, replicated commit log service.

1. Building Realtime data pipeline with
Apache Kafka
Nagarajan
Developer, ThoughtWorks

2. A pipeline?

3. Data pipeline
● Data flow between different systems/components
● Loosely coupled
● Automated
● Scalable
● Error recovery

4. Usecase
Image you’ve a web application, and you want to see number of visitors per page
on different time durations

9. Characteristics required
● High throughput ingestion
● Fault tolerant storage
● Highly available
● Scalable
● Support for concurrent processing and ordering guarantee

10. Enter Kafka
Apache Kafka® is a distributed streaming platform that:
● Publishes and subscribes to streams of records, similar to a message queue
or enterprise messaging system.
● Stores streams of records in a fault-tolerant durable way.
● Helps to process streams of records as they occur.
Kafka is a distributed, partitioned, replicated commit log service. It provides the
functionality of a messaging system with unique design.

11. Kafka APIs

12. Messaging system
● Queue
● publish-subscribe

13. Topics & Partitions

14. Partition and message ordering
Each partition is an ordered, immutable sequence of records that is continually
appended to a structured commit log
Records in the partitions are each assigned a sequential id number called the
offset that uniquely identifies each record within the partition
How does partition assigned?

15. Data retention
● Log retention by time
○ log.retention.ms (minutes/hours)
● Log retention by size
○ log.retention.bytes
● Log segments
○ log.segment.bytes
○ log.segment.ms
● Log compaction

16. Log compaction

17. Broker, Cluster and Zookeeper

18. Replication
● Replica types
○ Leader replica
○ Follower replica
● In Sync Replicas(ISR)
○ min.insync.replicas
● Replication factor

19. Producer
● acks
● Batching
○ batch.size
○ linger.ms
● Send Async
● Load balancing

20. Producer flow

21. Consumer and Consumer groups

22. Consumer and Consumer groups
● Push vs Pull
● Consumer position - offset maintenance
○ Auto commit
○ __consumer_offsets
● Replay
● Partition assignment
○ Range
○ Round robin

23. Delivery semantics
● At most once
● At least once
● Exactly once
○ Idempotency (exactly once produce)
○ Transactions (end to end exactly once) - only with Kafka streams

24. Schemas
● Decouple Producers and Consumers
● Avro format
● Schema Registry
● Evolution and compatibility

25. Fire fighting stories
● Consumer offsets retention
● Increase number of partitions
● Record without key

26. Resources:
https://www.confluent.io/resources
https://kafka.apache.org/

27. Feedback
bit.ly/geeknight_cbe

28. Advanced
● Security
● Configuration
● Monitoring
● Transactions
● Kafka connect
● Schema registry
● Kafka streams
● KSQL