Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.

1. Developing Frameworks
for Apache Mesos

2. Joe Stein
CEO of Elodina http://www.elodina.net/ a big data as a service platform
built on top open source software. The Elodina platform enables
customers to analyze data streams and programmatically react to the
results in real-time. We solve today’s data analytics needs by providing
the tools and support necessary to utilize open source technologies.
As users, contributors and committers, Elodina also provides support for
frameworks that run on Mesos including Apache Kafka, Exhibitor
(Zookeeper), Apache Storm, Apache Cassandra and a whole lot more!
LinkedIn: http://linkedin.com/in/charmalloc
Twitter : @allthingshadoop

3. ◉What goes on Mesos?
◉Framework = (Scheduler + Executor)
◉What does it look like without a
scheduler?
◉We can do better using a scheduler!
◉Schedulers working together.
◉Framework API & Examples.
Overview

4. What goes on Mesos?

6. Many, many things

7. Anything can be run on Mesos with Marathon or Aurora
https://mesosphere.github.io/marathon/ http://aurora.apache.org/

8. Framework = (Scheduler + Executor)

10. Scheduler

11. Executors

12. mesos/kafka
https://github.com/mesos/kafka

13. Scheduler
◉ Provides the operational automation for a Kafka Cluster.
◉ Manages the changes to the broker's configuration.
◉ Exposes a REST API for the CLI to use or any other
client.
◉ Runs on Marathon for high availability.
◉ Broker Failure Management “stickiness”
Executor
◉ The executor interacts with the kafka broker as an
intermediary to the scheduler
Scheduler & Executor

14. CLI & REST API
◉ scheduler - starts the scheduler.
◉ add - adds one more more brokers to the cluster.
◉ update - changes resources, constraints or broker properties one or more
brokers.
◉ remove - take a broker out of the cluster.
◉ start - starts a broker up.
◉ stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh
help stop)
◉ rebalance - allows you to rebalance a cluster either by selecting the brokers
or topics to rebalance. Manual assignment is still possible using the Apache
Kafka project tools. Rebalance can also change the replication factor on a
topic.
◉ help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}

15. Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1
./kafka-mesos.sh start 1000..1019

16. What does it look like
without a scheduler?

17. without a scheduler

18. without a scheduler

19. without a scheduler

20. without a scheduler

21. without a scheduler

22. We can do better
using a scheduler

23. with a scheduler

24. with a scheduler

25. with a scheduler

26. with a scheduler

27. with a scheduler

28. Schedulers working together

32. Framework API & Examples

33. Mesos Protos
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto
Everything is good to understand but here is a good place to start
◉ FrameworkInfo
◉ TaskInfo
◉ TaskState
◉ MasterInfo
◉ SlaveInfo

34. Mesos Framework Development Guide
http://mesos.apache.org/documentation/latest/app-framework-development-guide/
◉ Scheduler API
registered, reregistered, disconnected
resourceOffers, offerRescinded, statusUpdate, frameworkMessage
slaveLost, executorLost, error
◉ Executor API
registered, reregistered, disconnected
launchTask, killTask, frameworkMessage

35. Task Reconciliation
http://mesos.apache.org/documentation/latest/reconciliation/
It is the responsibility of Mesos (scheduler driver / Master) to
ensure that the framework is notified when a disconnection,
and subsequent (re-)registration occurs. At this point, the
scheduler should perform task state reconciliation.

36. Language Bindings
◉ c++ →
https://github.com/apache/mesos/tree/master/src/examples
◉ python →
https://github.com/apache/mesos/tree/master/src/examples/python
https://github.com/wickman/pesos
◉ java →
https://github.com/apache/mesos/tree/master/src/examples/java
http://mesos.apache.org/api/latest/java/
https://github.com/groupon/jesos
◉ go →
https://github.com/mesos/mesos-go
◉ clojure →
https://github.com/dgrnbrg/clj-mesos
https://github.com/pyr/mesomatic
◉ scala →
https://github.com/mesosphere/scala-sbt-mesos-framework.g8

37. Rendler
A rendering web crawler for Apache Mesos.
https://github.com/mesosphere/RENDLER

38. Questions?
Joe Stein
http://www.elodina.net