How many connects can you run in a single cluster? Disney + Hotstar runs over 10 different connect clusters with over 2000+ connectors. In this talk, we share our experience of running Kafka connect at scale. We will walk through our decisions of using one cluster vs many and how the improvements in the connect ecosystem like incremental rebalancing have allowed us to scale to thousands of connects. We will also discuss challenges with scaling up & down connect workers while keeping the ecosystem stable & present a wishlist of the missing features in this distributed task framework.

1. How much can you connect?

2. Agenda
● Introduction
● Why we love Connect?
● Paradigms of Connect Clusters
● Learnings
● Wishlist

3. Disney + Hotstar
● Leading OTT Platform
● Growing SEA presence
○ ID, SG, MY, TH (Coming Soon!)
○ Localised experience
● VoD, Live TV, Live Sports
● 25.3 Mn → Highest # of Concurrent viewers

4. Kafka @ D+H
● Central Nervous System
○ Analytics
○ Inter-Service Communication
○ Distributed Workflow Management
○ Alerting, Logging, Monitoring
○ User Engagement

5. Kafka Connect @ D+H
> 2000
Connectors
● Higher # of Sink ( ~2k)
● Few Source (20-30)

6. Kafka Connect @ D+H
> 2000
Connectors
● Higher # of Sink ( ~2k)
● Few Source (20-30)
10-15
Connect Clusters
● Auto-scaling based on
traffic
● Traffic + Use case
sharding

7. Kafka Connect @ D+H
> 2000
Connectors
● Higher # of Sink ( ~2k)
● Few Source (20-30)
10-15
Connect Clusters
● Auto-scaling based on
traffic
● Traffic + Use case
sharding

8. We ❤ Connect
Simple, flexible, extensible

9. Configuration not code
● Add/Remove pipelines
● Scale-up & down
{
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"s3.region": "us-east-1",
"topics.dir": "fanned-raw-events/WATCHED_VIDEO_EVENTS",
"flush.size": "1000000000",
"timezone": "GMT",
"tasks.max": "200",
"s3.part.size": "5242880",
"transforms": "Sampler,Decrypter",
"locale": "en",
"format.class": "io.confluent.connect.s3.format.json.JsonFormat",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"s3.bucket.name": "xxxxxxxxxxx",
"partition.duration.ms": "60000",
"schema.compatibility": "NONE",
"topics.regex": "prod.knol.watched_video",
"key.converter.schemas.enable": "false",
"s3.compression.type": "gzip",
"name": "warehouse-sink-watched_video",
"value.converter.schemas.enable": "false",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"path.format": "'cd'=YYYY-MM-dd/'hr'=HH",
"timestamp.extractor": "Wallclock",
"rotate.schedule.interval.ms": "300000"
}

10. No more tuning
● Leave client configuration headache out
● Upgrade faster
● Get failure handling out of the way

11. Fan out quickly

12. Fan out quickly

13. Flexible Building Blocks
● Single Message Transform(SMT)
○ Masking/Filtering of PII
○ Sampling of data
○ Schema Validation & Enforcement
● Chaining of different transforms
● Lambda style architecture
○ Reusable
○ Easy to implement

14. Extensible
● Code/deploy custom connectors quickly
● Don’t look under the hood

15. Connect Deployment Paradigms

16. Terminology

17. 1 Connector per cluster

18. All Connectors in 1 cluster

19. Connect Paradigms: Summary
✅ Isolation
✅ No rebalance impact
❌ Not efficient
❌ Too many Kafka topics for
connect management
✅ Efficient
✅ Lesser Topics
❌ Rebalance Storms

20. Connect Paradigms: Summary
✅ Isolation
✅ No rebalance impact
❌ Not efficient
❌ Too many Kafka topics for
connect management
✅ Efficient
✅ Lesser Topics
❌ Rebalance Storms

21. Learnings

22. Separate clusters
Traffic Patterns
● Optimize for scaling of the cluster
○ Uniform byte rate across tasks
● Metadata vs Data
○ Separate S2S/CDC data from other
flows

23. All tasks are not equal

24. Separate Clusters
Use-case
● # of configuration changes
● Resiliency to restarts & rebalances

25. Restart those connects
● Transient failures occur all the time
● Monitor JMX
● Focus on outcome, not status
○ Consumer-lag vs task-count
● Periodically restart

26. Don’t scale too slow
● More steps of scaling == More rebalances
● Ladder based scaling
○ Scale up/down slow in large increments

27. Don’t scale too slow

28. Wishlist
● YARN like support
○ Priority lanes
○ Elastic Capacity
● Non-Kafka based config & offset sources
● Task/Worker ratio

29. Thank you
Bhavesh Raheja
Twitter: @bhaveshraheja
tech.hotstar.com