5. Kafka without Security is RISKY
5 disastrous scenarios
1. Read all your data
2. Write to any topic and break your consumers
3. Intercept and read plaintext network packets
4. Delete all your Kafka data in one command without SSH
5. Kafka Connect? Database Credentials are in a Kafka Topic, plaintext
6. You need Kafka Security
If you intend to make Kafka a cornerstone of your infrastructure
8. Kafka Security in three words
Encryption
Authentication
Authorization
9. Encryption in Kafka
• SSL encryption = secure communications
• Similar to HTTPS
Super secret
message
Kafka Client
(producer / consumer )
Kafka Brokers
Port 9093 - SSL
aGVsbG8gd29
ybGQgZWh…
Encrypted data
Kafka Client
(producer / consumer )
Kafka Brokers
Port 9092 - PLAINTEXT
10. SSL, Concretely?
• Create a Certificate Authority (CA)
• Generate certificates for your brokers, sign them
• Make sure your broker and clients trust the CA Root.
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=serversecret
ssl.key.password=serversecret
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=serversecret
11. SSL in the Real World
• SSL lowers the performance of your brokers
• You lose the zero-copy optimization
• Kafka heap usage increases
• CPU usage increases
• SSL only allows to encrypt data in flight
• Data at rest sits un-encrypted on Kafka Disk
12. What about end-to-end encryption?
• Closed source: Apple
• Open source:
• https://github.com/Quicksign/kafka-encryption
• https://github.com/nucypher/kafka-oss
• POC in progress at DataCumulus
Producer
Kafka
PLAINTEXT
Consumer
Encrypted
data
Encrypted
data
encrypt data decrypt data
14. Authentication in Kafka
• Clients need to have and prove their identity
• ~= Login (username / password or token)
Kafka Client Kafka Broker
Authentication data
Verify authentication
Client is authenticated
15. 99 Forms Of Authentication
But Easy Ain’t One
• SSL Authentication: two way client authentication
• SASL (Simple Authentication and Security Layer):
• SASL/GSSAPI (Kerberos) – v0.9.0.0+ - Enterprises (Microsoft AD)
• SASL/PLAIN – v0.10.0.0+ - Passwords hardcoded in broker
• SASL/SCRAM-SHA-256/512 – v0.10.2.0+ - Passwords in Zookeeper (secure it)
• SASL/OAUTHBEARER – v2.0+ - Leverage OAuth 2
• Write your own (contribute back!)
• Extend SASL/PLAIN and SASL/SCRAM with KIP-86 (change credentials store)
• Real world advice:
choose the authentication mechanism you already have in your enterprise
16. Take-aways from the battlefield
• SSL authentication makes it really hard to revoke authentication
• SASL (Simple Authentication and Security Layer) is not simple (YMMV)
• Kerberos is by far the hardest to setup right. Errors are cryptic
• This is the most challenging part of the Kafka security journey
17. Authentication in Kerberos, concretely?
1. Create Kerberos or use Active Directory
2. Ensure Kafka servers have correct CNAME & hostname
3. Generate credentials for the brokers
4. Generate KeyTabs for the brokers from the credentials
5. Create a JAAS file:
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/tmp/kafka.service.keytab"
principal="kafka/<<KAFKA-SERVER-INTERNAL-DNS>>@KAFKA.SECURE";
};
18. Authentication in Kerberos, concretely?
Continued…
• Start Kafka and use java options to reference JAAS file
• Add properties to Kafka:
• Start Kafka
• Pray !
advertised.listeners=SASL_SSL://<<KAFKA-SERVER-DNS>>:9094
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
19. Real-World Tips: Authentication
• Turn on DEBUG log during setup!
• Go slow
• Ensure CNAME and PTR records are correct
• Scrape and repeat can sometimes solve issues
• Automate
21. Authorisation in Kafka
• Kafka knows our client’s identity
• + Authorization rules:
• ”User alice can read topic finance”
• ”User bob cannot write topic trucks”
• = Security
• ACL (Access Control Lists) have to be maintained by administrators
22. ACLs, where are they?
• Default:ACLs are stored in Zookeeper
• Must secure Zookeeper (network rules or authentication)
• OR write your own authorizer (AD, LDAP, a database, Kafka…)
25. Managing ACLs at Scale?
• Look into Kafka Security Manager
(https://github.com/simplesteph/kafka-security-manager )
26. Real World Tips on ACLs
• Authorisation denials will be logged as INFO in the Kafka log.
• Define your broker as super users
• Careful with: allow.everyone.if.no.acl.found=true
• ACLs can be applied to:
• Topics: Create, Read, Describe,Write, etc…
• Groups: Read,Write, Describe
• Cluster: DescribeConfigs,AlterConfigs, Create
• Wildcards are supported in Kafka 2.0! (useful for Kafka Streams)
31. Kafka Client Security
is the real challenge
• Technical Challenge:
• Java Clients: easy
• Non Java Clients: please use a client that wraps librdkafka
• People Challenge:
• Kafka Administrator: I’m a security guru! But I don’t want to secure all the apps
• Kafka Developer: wt* is security?
32. Client Security in Java
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/home/kafka/ssl/kafka.client.truststore.jks
ssl.truststore.password=clientpass
sasl.jaas.config='com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/client.service.keytab"
principal="clientusername";'
• It’s not fun
• It’s not easy
• It’s error prone
33. Kafka Clients are not tailored to your
security needs
• Observation 1: Default Kafka clients have every options for security
• Observation 2: Your enterprise will only have one security setup
• Observation 3: Every client security configuration will look the same
• Take-away: don’t use the default Kafka clients
34. Real World Advice #1
Distribute your own wrapped Kafka Clients
• Developers love nice APIs:
• Standardized applications
• No copy and paste errors
• Centralized debugging
• Reduced learning curve for devs
new MyCorpKafkaProducer(bootstrapServers, keySerializer, valueSerializer)
.withSSL(sslEnabled, pathToTrustStore)
.withAuth(authEnabled, pathToKeyTab, usernameOrPrincipal)
.withSchemaRegistry(url)
.withExtraProperties(properties)
.build()
35. Real World Advice #2
Create a Kafka Client Base Docker Image
Security at scale goes hands in hands with consistency.
• Embed modified Java Truststore
• Standard Retrieval of SSL certificates
• Standard Retrieval of Credentials fromVault / Secure Store
• Kafka environment switches, Security switches
• Bootstrap Server Discovery & Schema Registry Discovery
• Extend to Kafka Connect & Schema Registry
36. Real World Advice #3
Make a checklist before going to prod
• What’s the application username?
• Are all ACLs listed and created?
• Is the application using the MyCorp Kafka clients?
• Is the application running in the standardized Docker Container?
• Are quotas defined for this application?
• Is the application monitored?
• …Check? Release!