Kobi Hikri (Independent Software Architect and Consultant): Kobi provides a short intro to Kafka Connect, and then shows an actual code example of developing and dockerizing a custom connector.

1. Introducing Kafka
Connect and
ImplementingCustom
Connectors
Kobi Hikri
Kafka Connect

2. Agenda
 What (is Kafka Connect)
 The World of Data
 Why (do we need it) Kafka Connect
 When (will we consider using) Kafka Connect
 How Kafka Connect
 Setting up the infrastructure
 Interacting with Kafka Connect
 Implementing a custom source connector
 Working with connectors
 What will not be covered in this talk (will be covered per request)
 Kafka ConnectTransformations
 Working with custom schemas

3. What is Kafka
Connect
 A Kafka integration tool
 Allows importing data into Kafka
 And exporting data out of Kafka
 Runs as a cluster of one or more system processes
 Each such process is referred to as a Worker
 Workers are grouped together in Worker Groups
 Connector Instances are managed by theConnect cluster
 Tasks are configured by connectors and are distributed across workers
 Data in being pulled by SourceTasks (configured by Source Connectors)
 Or Pushed by Sink Tasks (configured by Sink Connectors)
Worker
Worker
Worker
Worker
Worker Group Worker Group

4. TheWorld of Data
Dataisscatteredallover!
AJigsawandplentyofglueareneeded

5. Why KafkaConnect
 In one word
 Integration
 Of different data oriented technologies
 Of a multitude of changing APIs and communication protocols
 In more than one word
 Fault tolerant, scalable Integration with various technologies

6. When Kafka
Connect
 Implementing a Kafka Producer or Consumer within the
desired data source / target is impossible or expensive (e.g.
the source code is unavailable)
 Even when we can modify the source code of the source /
target system – but prefer to have a scalable and fault tolerant
integration with it (e.g. not to be limited in bandwidth by the
direct connection between the technology and our Kafka
cluster).

7. How KafkaConnect
Demo
Time
Docker
Compose
Kafka
Connect
Rest API
Custom
Source
Connector