This document discusses Kafka on Mesos, which allows Kafka to elastically scale on the Mesos cluster. It provides a quick introduction to Mesos and Kafka. The Kafka on Mesos project aims to make Kafka elastic by allowing smart broker assignment and configuration changes, rolling restarts, and scaling the cluster up and down both automatically and manually. It includes a scheduler and executor that manage the Kafka brokers. The CLI and REST API allow adding, updating, removing, starting, and stopping brokers as well as rebalancing topics. An example demonstrates launching 20 brokers in seconds on Mesos. Finally, it notes that Kafka is available on DCOS for deployment and management on Mesosphere DC/OS.

1. Kafka on Mesos
https://github.com/mesos/kafka

2. Joe Stein
CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing
on the support & maintenance of third party open source software (like
Mesos frameworks) as well as its own open source products & SaaS
solutions.
Joe Stein is also the Founder and Principal Consultant of Big Data Open
Source Security http://stealth.ly and has been working for the last couple
of years on implementing and assisting organizations with their Kafka,
Mesos, Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data
systems.
Prior to this, Joe Stein was responsible for building out a platform that
ingested and processed the analytics for 6 billion unique mobile devices.

3. ◉ Quick intro to Mesos
◉ Quick intro to Kafka
◉ Making Kafka elastic with
Mesos
Overview

4. Quick intro to Mesos

5. Static partitioning

6. Static partitioning

7. Static partitioning

8. Static partitioning

9. Better option

10. Data Center Operating System

12. Mesos

13. Quick intro to Kafka

14. Not Streaming Data

15. Kafka decouples data pipelines

16. Making Kafka elastic with Mesos

17. Goals we set out with
● smart broker.id assignment.
● preservation of broker placement (through constraints
and/or new features).
● ability to-do configuration changes.
● rolling restarts (for things like configuration changes).
● scaling the cluster up and down with automatic,
programmatic and manual options.
● smart partition assignment via constraints visa vi
roles, resources and attributes.

18. mesos/kafka
https://github.com/mesos/kafka

19. Scheduler
● Provides the operational automation for a Kafka Cluster.
● Manages the changes to the broker's configuration.
● Exposes a REST API for the CLI to use or any other
client.
● Runs on Marathon for high availability.
Executor
● The executor interacts with the kafka broker as an
intermediary to the scheduler
Scheduler & Executor

20. CLI & REST API
● scheduler - starts the scheduler.
● add - adds one more more brokers to the cluster.
● update - changes resources, constraints or broker properties one or more
brokers.
● remove - take a broker out of the cluster.
● start - starts a broker up.
● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh
help stop)
● rebalance - allows you to rebalance a cluster either by selecting the brokers
or topics to rebalance. Manual assignment is still possible using the Apache
Kafka project tools. Rebalance can also change the replication factor on a
topic.
● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}

21. Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1
./kafka-mesos.sh start 1000..1019

22. Kafka is available on DCOS
https://mesosphere.com/product/
dcos install kafka
dcos kafka help
Mesosphere DCOS

23. Questions?
Joe Stein
http://www.elodina.net