"Kafka Connect is the highway to our central nervous system for streaming data into Kafka. This highway has allowed us to connect hundreds of external services without the need to write code for each individual integration. Kafka Connect is based on the concept of having Connectors and tasks that do the work for us. There’s two fundamental questions that come up when it comes to our highway: • How is this highway made? How are connectors created/started? • How do we make sure there’s no traffic jams? Also known as, • How are connectors evenly distributed across nodes? In this presentation, we will answer the above questions and cover the following topics: • Breaking down how a Connector/task is created • Breaking down each Kafka Connect protocol(eager, compatible, sessioned) • Walk through rebalances for each protocol • Pros/cons of each protocol"

1. Deep Dive in Connect Protocol
@cdpop_
Catalin Pop

2. ❏ High level overview of Connect
❏ Breaking down creation of Connectors
❏ Breaking down Connect Protocols
❏ Walk through rebalance
❏ Pros/Cons of each protocol
@cdpop_

3. High level overview of Connect
@cdpop_

4. High level overview of Connect
@cdpop_

5. High level overview of Connect
@cdpop_

6. High level overview of Connect
@cdpop_

7. High level overview of Connect
@cdpop_

8. High level overview of Connect
@cdpop_

9. High level overview of Connect
@cdpop_

10. High level overview of Connect
@cdpop_

11. High level overview of Connect
@cdpop_

12. ✓ High level overview of Connect
❏ Breaking down creation of Connectors
❏ Breaking down Connect Protocols
❏ Walk through rebalance
❏ Pros/Cons of each protocol
@cdpop_

13. Connector Creation
@cdpop_

14. Connector Creation
@cdpop_

15. Connector Creation
@cdpop_

16. Connector Creation
@cdpop_

17. Connector Creation
@cdpop_

18. Connector Creation
@cdpop_

19. Connector Creation
@cdpop_

20. Connector Creation
@cdpop_

21. Connector Creation
@cdpop_

22. Connector Creation
@cdpop_

23. Connector Creation
@cdpop_

24. Connector Creation
@cdpop_

25. Connector Creation
@cdpop_

26. Connector Creation
@cdpop_

27. Connector Creation
@cdpop_

28. Connector Creation
@cdpop_

29. Connector Creation
@cdpop_

30. Connector Creation
@cdpop_

31. Connector Creation
@cdpop_

32. Connector Creation
@cdpop_

33. Connector Creation
@cdpop_

34. Connector Creation
@cdpop_

35. Connector Creation
@cdpop_

36. Connector Creation
@cdpop_

37. ✓ High level overview of Connect
✓ Breaking down creation of Connectors
❏ Breaking down Connect Protocols
❏ Walk through rebalance
❏ Pros/Cons of each protocol
@cdpop_
@cdpop_

38. Breaking down connect protocols
@cdpop_

39. Breaking down connect protocols
@cdpop_

40. Eager Protocol
● Revoke all Connectors
● Perform round-robin assignment
● Send Connector assignment to queue
● Workers pickup assignment
● Start assigned connectors
@cdpop_

41. Compatible Protocol
● Revoke connectors when the worker left the group or connector/task change
● Generate connector assignment only for the revoked connectors
● Send Connector assignment to queue
● Workers pickup assignment
● Start assigned connectors
@cdpop_

42. Sessioned Protocol
● Setup Session key for encrypted authentication
● Revoke connectors when the worker left the group or connector/task change
● Generate connector assignment only for the revoked connectors
● Send Connector assignment to queue
● Workers pickup assignment
● Start assigned connectors
@cdpop_

43. ✓ High level overview of Connect
✓ Breaking down creation of Connectors
✓ Breaking down Connect Protocols
❏ Walk through rebalance
❏ Pros/Cons of each protocol
@cdpop_

44. Walk through rebalance
@cdpop_

45. Walk through rebalance
@cdpop_

46. Walk through rebalance
@cdpop_

47. Walk through rebalance
@cdpop_

48. Walk through rebalance
@cdpop_

49. Walk through rebalance
@cdpop_

50. Walk through rebalance
@cdpop_

51. Walk through rebalance
@cdpop_

52. Walk through rebalance
@cdpop_

53. Walk through rebalance
@cdpop_

54. Walk through rebalance
@cdpop_

55. Walk through rebalance
@cdpop_

56. Walk through rebalance
@cdpop_

57. Walk through rebalance - Eager
@cdpop_

58. Walk through rebalance - Eager
@cdpop_

59. Walk through rebalance - Eager
@cdpop_

60. Walk through rebalance - Eager
@cdpop_

61. Walk through rebalance - Eager
@cdpop_

62. Walk through rebalance - Compatible/Sessioned
@cdpop_

63. Walk through rebalance - Compatible/Sessioned
@cdpop_

64. Walk through rebalance - Compatible/Sessioned
@cdpop_

65. Walk through rebalance - Compatible/Sessioned
@cdpop_

66. ✓ High level overview of Connect
✓ Breaking down creation of Connectors
✓ Breaking down Connect Protocols
✓ Walk through rebalance
❏ Pros/Cons of each protocol
@cdpop_

67. Pros/Cons of protocols
Eager
Pro
Evenly
distributed
Con
Not secured
Stops all
connectors
@cdpop_

68. Pros/Cons of protocols
Eager
Pro
Evenly
distributed
Con
Not secured
Stops all
connectors
Compatible
Pro
Incremental
rebalance
Con
Not secured
Uneven
distribution
@cdpop_

69. Pros/Cons of protocols
Eager
Pro
Evenly
distributed
Con
Not secured
Stops all
connectors
Sessioned
Pro
Secure
Incremental
rebalance
Con
Uneven
distribution
Compatible
Pro
Incremental
rebalance
Con
Not secured
Uneven
distribution
@cdpop_

70. Thank you
@cdpop_