Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos

Testing SMTs?
Testcontainers to the rescue!
Fábio Sequeira | Mafalda Santos
Marionete

2
© 2023 Marionete Limited
• Kafka Connect is a tool for scalability and reliable data transmission between Apache Kafka and other data systems.
• Kafka Connectors are ready-to-use components useful for importing and exporting data between Kafka topics and external
systems (e.g. databases).
Testing SMTs
Kafka Connect & Connectors
Source Connector Sink Connector
Data Source Data Sink
Kafka

3
• Single Message Transforms (SMTs) are used to transform message values and keys.
• Just like the connectors, there are ready-made and easy-to-use SMTs available.
Testing SMTs
SMTs
Source Connector Sink Connector
Data Source Data Sink
Kafka
SMT SMT

4
There are plenty of SMTs available but…
Testing SMTs

5
There are plenty of SMTs available but…
Testing SMTs
Sometimes they are not enough.

6
We need to build
CUSTOM SMTs!
Testing SMTs

7
SMTs are relatively easy to build…
Testing SMTs

8
SMTs are relatively easy to build…
Testing SMTs

9
Testing SMTs
Unit Tests Limitations
Data Source
Kafka Cluster
Kafka Connect
Connector
Instance
SMTs
Converter
Connect
Record

10
Testing SMTs
Data Source
Kafka Cluster
Kafka Connect
Connector
Instance
SMTs
Converter
Connect
Record
Unit Tests
?
?
?

11
Testing SMTs
• Hard to understand how the Connector reads the value types coming from a DB:
• Often got data in different formats or patterns than we expected
• Different connectors could “read” data differently

12
Testing SMTs
• Certain DB-specific types are very difficult to define in unit tests:
• SQL Server types: Datetime2? Datetimeoffset?

13
Testing SMTs
• Certain DB-specific types are very difficult to define in unit tests:
• SQL Server types: Datetime2? Datetimeoffset?
• Difficult to map/identify specific topic schema fields:
• Protobuf schemas: optional/oneof fields

14
How to complement the SMTs tests with a more robust and effective test?
Testing SMTs

15
How to complement the SMTs tests with a more robust and effective test?
Testing SMTs

Testcontainers
to the rescue!

17
Testcontainers
• Set up, configure, and run Docker
containers
• Simplify integration testing
• Available in multiple languages
• (We used Java)

18
Containers for SMT Testing
• The Testcontainers library has a module for Kafka
• Example:
KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.3.2")
)
• There are also modules for various types of databases.
• For our use cases, we used a Container object based on the Testcontainers MS SQL Server module.
• But we needed more than that…

19
Custom Testcontainer Library
KafkaContainer SchemaRegistryContainer ConnectContainer MsSqlServerContainer

20
SchemaRegistryContainer
public final class SchemaRegistryContainer extends GenericContainer<SchemaRegistryContainer> {
public SchemaRegistryContainer(DockerImageName image) {
super(image);
this.addExposedPorts(8081);
this.withEnv("SCHEMA_REGISTRY_HOST_NAME", this.getHost());
}
public SchemaRegistryContainer withKafka(KafkaContainer kafkaContainer) {
this.withNetwork(kafkaContainer.getNetwork());
this.withEnv(
"SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS",
"PLAINTEXT://" + kafkaContainer.getNetworkAliases().get(0) + ":9092"
);
return this;
}
}

21
SchemaRegistryContainer (cont)
public SchemaRegistryContainer setupSchemaRegistryContainer(
KafkaContainer kafkaContainer,
String alias,
String confluentVersion
) {
return new SchemaRegistryContainer(
DockerImageName
.parse("confluentinc/cp-schema-registry:" + confluentVersion))
.withNetworkAliases(alias)
.withKafka(kafkaContainer);
}

22
ConnectContainer
public final class ConnectContainer extends GenericContainer<ConnectContainer> {
public ConnectContainer(ImageFromDockerfile image) {
super(image);
this.addExposedPorts(8083);
this.withEnv("CONNECT_GROUP_ID", "testcontainer-connect-group");
this.withEnv("CONNECT_CONFIG_STORAGE_TOPIC", "connect-config");
this.withEnv("CONNECT_OFFSET_STORAGE_TOPIC", "connect-offsets");
this.withEnv("CONNECT_STATUS_STORAGE_TOPIC", "connect-status");
this.withEnv("CONNECT_REST_ADVERTISED_HOST_NAME", this.getHost());
this.withEnv(
"CONNECT_PLUGIN_PATH",
"/usr/share/java, /usr/share/confluent-hub-components/"
);
// ...
}
public ConnectContainer withKafka(KafkaContainer kafka, SchemaRegistryContainer registry) {
this.withNetwork(kafka.getNetwork());
this.withEnv(
"CONNECT_BOOTSTRAP_SERVERS",
"PLAINTEXT://" + kafka.getNetworkAliases().get(0) + ":9092”
);
this.dependsOn(kafka, registry);
return this;
}
}

23
ConnectContainer (cont)
public ConnectContainer setupConnectContainer(
KafkaContainer kafkaContainer,
SchemaRegistryContainer registryContainer,
String alias,
String confluentVersion
) {
return new ConnectContainer(
new ImageFromDockerfile().withDockerfileFromBuilder(
dockerfileBuilder -> dockerfileBuilder
.from("confluentinc/cp-kafka-connect:" + confluentVersion)
.run("/bin/bash", "-c", "confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0")
.build()
)
)
.withKafka(kafkaContainer, registryContainer)
.withNetworkAliases(alias);
}

24
Custom Library Example Helper Methods
• createTopic() • registerSchema() • installSMT()
• registerConnector()
• runSQLFile()
KafkaContainer SchemaRegistryContainer ConnectContainer MsSqlServerContainer

25
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer

26
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Custom SMT
Generate jar

27
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Custom SMT

28
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
Custom SMT

29
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
Custom SMT

30
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Custom SMT

31
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Consumer
Custom SMT

32
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Consumer
test-source-connector
Custom SMT

33
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Consumer
Custom SMT
{
"name": "test-source-connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"tasks.max": "1",
"key.converter":
"org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema:8081",
"value.converter.auto.register.schemas": "false",
"topic.prefix": "output-topic",
"connection.url":
"jdbc:sqlserver://mssql:1433;databaseName=TestDB;(...)",
"table.whitelist": "Input_Table",
// ...
"transforms": "myCustomSMT",
"transforms.myCustomSMT.type": "org.example.MyCustomSMT$Value",
"transforms.myCustomSMT.targetFields": "field1,field2"
}
}

34
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Consumer
Input records
Custom SMT

35
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
Input table
output-topic
output-topic-value
Consumer
Input records
Compare retrieved records
with expected records
Custom SMT

36
Test Setup – Sink Connectors
KafkaContainer
SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer MsSqlServerContainer
Output table
input-topic
input-topic-value
Producer
test-sink-connector
Compare retrieved records
with expected records

37
Thank you!

Contacts
www.marionete.co.uk
FÁBIO SEQUEIRA
fabiosequeira
MAFALDA SANTOS
mafaldajsantos
solutions@marionete.co.uk
@marionete_io
https://www.linkedin.com/company/marionete
Core Technology
Specialist
Core Technology
Specialist

Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos

Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos

Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos

Recomendados

Más contenido relacionado

Similar a Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos

Similar a Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos(20)

Más de HostedbyConfluent

Más de HostedbyConfluent(20)

Último

Último(20)

Testing SMTs? Testcontainers to the Rescue! with Fábio Sequeira & Mafalda Santos