Pere Urbon-Bayes, Confluent, Software Engineer Have you ever wondered, what motivates people? According to many authors, autonomy is a key factor to build a sustainable and motivated team. This is as well a core principle of the devops. If you are in the quest of building a self-service kafka based platform where your users can manage topics, acls and rbac by themself, but still keep change control and a verification process? Gitops and Kafka Topology Builder is here to help you. In this talk we're going to present and guide you towards the steps necessary to build up a self-service Apache Kafka using Gitops and Kafka Topology Builder, including a few war stories from people using it in production. By the end of this talk, you will take home an automatic Jenkins pipeline, that will make your users autonomous and give you more time to focus on less bureaucratic work. https://www.meetup.com/Mexico-Kafka/events/276039165/

1. Kafka Topology Builder,
bringing autonomy to the
Confluent Platform

2. Who am I ?
Software Engineer at Confluent, previously a Professional Services
Working around data in the IT industry (and university) for more than 15 years
Handball and Lego fan
Living in Berlin, Germany since 2011
2

3. What is motivation?
3
I have a few questions for you?
● What do you recall from the best team you
have ever been?
● How do you think teams keep a high level of
motivation?
● What do you think usually works best?

4. What motivates people the best
● Autonomy: Our desire to be self directed. It
increases engagement over compliance.
● Mastery: The urge to get better skills.
● Purpose: The desire to do something that has
meaning and is important.
4

5. How do we get that in Apache Kafka
and Confluent Platform?
Building self sufficient Confluent Platform deployments

6. Building autonomous and self sufficient Kafka
teams
Common questions teams deploying Confluent Platform face with:
● How can a user request the topics they need for their applications?
● How can an ops team ensure ACLs/RBAC is in place for each application?
● How can schemas be structure migrated across environments?
Teams usually involved here: Development, Operations, Security and Audit, all
with different requirements and needs.
6

7. How can we solve this
methodologically?
Manual processes over full automation

8. @purbon
Common
questions for
KafkaOps

9. @purbon
Email always works, right?

10. @purbon
Or maybe try Jira?

11. @purbon
What happens to my logs?

12. @purbon
Automate all the things!

13. @purbon
A declarative method

14. Bringing automation for autonomy
Ok, we settled on automate all the things.. what do we need for this:
● A way for developers to describe what they need.
● A system that interpret users (developers) wishes and is able to apply them
automatically to the target platform.
● Automation in place to reduce unnecessary interactions.
14

15. Self service for Apache Kafka and Confluent
Platform
The Kafka Topology Builder and gitops can help automating this process with a
declarative, centralised and self service solution to maintain the day to day
operations in Confluent Platform.
One more step for a self service kafka
15

16. What does Kafka Topology Builder offer you?
Continuous delivery of changes to Apache
Kafka and Confluent Platform
Multitenancy out of the box
All changes are documented and
approved. Full control of change
management for audit
Declarative approach, users don’t need to
know low level of Kafka
Full development team autonomy, without
hampering operational control
16

17. In detail, what do I get with this process?
● API driven solution to integrate with CI/CD practises in house.
○ Support for on-prem cluster and Confluent Cloud.
● Topic management with:
○ creation, delete and update of specific configuration value
○ out of the box namespacing based on the project structure and data types.
● Multi tenancy enablement out of the box
● Access Control management with:
○ Predefined roles for Consumers, Producers, Kafka Connect Connectors and Kafka
Streams applications
○ Support for traditional ACLs and Confluent RBAC
○ Platform wide ACLs for Schema Registry, Kafka Connect, Control Center, etc..
● Schema Management when using Confluent Schema Registry.
NOTE: For all this futures, the tool managed automatic create, delete and update when
required.
17

18. To know more about the project
If you are interested to know more about the project, you can find more information in:
● Kafka Topology Builder Github: https://github.com/purbon/kafka-topology-builder
● Documentation: https://kafka-topology-builder.readthedocs.io/
● Releases: https://github.com/purbon/kafka-topology-builder/releases
18

19. Does anything changes if I do containers?
19
Professional Services

20. What is a Service Broker?
20

21. Kafka Service Broker Workflow on CloudFoundry
Service Instantiation
21
Cloud
Controller
Application Runtime
Kafka Service
Broker
1. My App needs access to a topic!
cf create-service confluent-kafka gold t1
2. create
topic
3. create topic
Success/Failure
response
Success /
Failure
response
Success/
Failure
response
MyApp
MyApp

22. Kafka Service Broker on CloudFoundry
Service Binding
22
Cloud
Controller
PWS Application
Runtime
Kafka Service
Broker
6. create user and
api key
7. set ACLs
4. bind topic to my app
cf bind-service my-app t1
5. bindRequest my-app, t1
8. return user, password
MyApp
9. Inject credentials as
environment variable
9. Read Credentials from
Environment upon startup
MyApp

23. Many are available, for Kubernetes and Cloud
Foundry
● A Kafka Service Broker for Cloud Foundry at BOSH
● Pivotal - Datastax: Cassandra Service Broker
● Service Broker for MongoDB Atlas
● Another Kafka Service Broker by 1123 (by our own Benedikt Linse)
23

24. Demo time….
24

25. Thanks! questions?