Making developers productive on Kafka requires giving self-service access. But even something as seemingly straightforward as Topic creation is not so easy, and in some cases can lead to catastrophe.
In this talk, we’ll share and demonstrate different approaches for developers to safely create Kafka Topics whilst sharing a few war stories of what can go wrong along the way.
Creating a Kafka Topic. Super easy? | Andrew Stevenson and Marios Andreopoulos, Lenses
1. Creating a Kafka Topic. Super easy?
Making developers productive on Kafka requires giving self-
service access. But even something as seemingly straightforward
as Topic creation is not so easy, and in some cases can lead to
catastrophe.
In this talk we’ll share and demonstrate different approaches for
developers to safely create Kafka Topics whilst sharing a few war
stories of what can go wrong along the way.
2. Creating a Topic
Super easy?
Andrew Stevenson
CTO
Marios Andreopoulos
Head of Eng. Ops
#KafkaSummit21
@lensesio
8. Partitions: How Kafka Scales
● Data between partitions need to be orthogonal
● Changing number of partitions would re-key new messages
● Too many partitions, more metadata, etc
9. Partitions and Retention
● Retention applies to partitions
● Retention of 10GB for 100 partitions means at least 1TB of disk space
● Log segment size can mean at least X TB of disk space
● More partitions, more metadata
10. Replication
● Kafka was created so the bottleneck is your network
● Too high replication may saturate the network
● Replication can affect delay between producing and consuming a message
● Low replication affects availability
14. Put guardrails to config
Control capacity
Enforce policies
Guard configs
Enable namespacing
15. Add metadata
Make it discoverable
Document what is this about
Classify topics with relevant content
16. Authorize access
Think of naming conventions ( you can’t change names )
Map names to Namespaces
Namespaces can drive your security and governance strategy (such as with ACLs)
18. Automate the process
“Topics-as-code”
Automate & CI/CD
Create your GitOps strategy
name: red_barons
replication: 1
partitions: 3
configs:
retention.ms: "1440000"
segment.bytes: "104857600"
description: "awesome topic"
tags: ["tag1", "tag2" ]
19. Options for how to govern topic creation
Free for all
Happy
developers, Ops
plan for bad days
Highly
constrained
Highly
constrained but
automated
100% Self-serve
ServiceNow,
Email, Jira,
Slack… backlogs
More like a PR...
Guardrails …
20. Tooling for Apache Kafka &
Kubernetes
Commercial offering for Kafka & Kubernetes
>1 Million
Downloads
>5000
Github Stars
21. A new way of working
with real-time data:
DataOps
A developer workspace
for building & operating
real time applications
on Kafka