Jointly presented at EMC World 2016 with Somik Behera of Mesosphere

1. 1© Copyright 2016 EMC Corporation. All rights reserved.

2. 2© Copyright 2016 EMC Corporation. All rights reserved. 2© Copyright 2016 EMC Corporation. All rights reserved.
INTRODUCTION TO
MESOS AND MESOSPHERE
MAY 2016

3. 3© Copyright 2016 EMC Corporation. All rights reserved.
• Somik Behera, DCOS Product Manager - Mesosphere
– Twitter: @strikesme
– Blog: http://strikesme.com
• David vonThenen, Developer Advocate - EMC {code}
– Twitter: @dvonthenen
– Blog: http://dvonthenen.com
PRESENTERS

4. 4© Copyright 2016 EMC Corporation. All rights reserved.
• What is Mesos?
• What will Mesos do for me?
• Who is Mesosphere?
• How EMC{code} fits with Mesos
• Demo
• Q&A
AGENDA

5. 5© Copyright 2016 EMC Corporation. All rights reserved. 5© Copyright 2016 EMC Corporation. All rights reserved.
WHAT IS MESOS?

6. 6© Copyright 2016 EMC Corporation. All rights reserved.
• From a resource perspective, it’s a cluster manager:
– Pools servers resources to be centrally managed as a single
unit
• From an application perspective, it’s a scheduler:
– Dispatches workloads to consume pooled resources
• Often described as a Data Center Operating System
(DCOS)
MESOS IS…

7. 7© Copyright 2016 EMC Corporation. All rights reserved.
• Master
– Manages Mesos Agents
– Enables fine-grained sharing of resources (CPU, RAM, …)
across frameworks in the form of resource offers.
• Zookeeper
– Enables highly reliable distributed coordination (cluster
state)
• Agent (aka Slave)
– The entity that will be running the task
MESOS COMPONENTS

8. 8© Copyright 2016 EMC Corporation. All rights reserved.
MESOS ARCHITECTURE

9. 9© Copyright 2016 EMC Corporation. All rights reserved.
EFFICIENT RESOURCE UTILIZATION

10. 10© Copyright 2016 EMC Corporation. All rights reserved.
WHO DOESN’T LIKE EFFICIENCY?

11. 11© Copyright 2016 EMC Corporation. All rights reserved.
• Generic
– Anything that can be run on a standard Linux shell
– Provides a mechanism for deploying applications packages
– Supports Docker workloads
• Framework
– A plug-in to Mesos that allows you to implement a sub-
scheduler
– specialize a workload by providing custom placement and
resource allocation
APPLICATION SUPPORT

12. 12© Copyright 2016 EMC Corporation. All rights reserved.
GENERIC APPLICATIONS

13. 13© Copyright 2016 EMC Corporation. All rights reserved.
MESOS FRAMEWORKS

14. 14© Copyright 2016 EMC Corporation. All rights reserved.
BRINGING EVERYTHING TOGETHER

15. 15© Copyright 2016 EMC Corporation. All rights reserved. 15© Copyright 2016 EMC Corporation. All rights reserved.
WHAT WILL MESOS DO FOR ME?

16. 16© Copyright 2016 EMC Corporation. All rights reserved.
• Bare metal x86 hardware
• VMs (your favorite hypervisor)
• Public, Private, Hybrid Cloud
MULTIPLE DEPLOYMENT VECTORS

17. 17© Copyright 2016 EMC Corporation. All rights reserved.
MESOS WILL RUN ON…

18. 18© Copyright 2016 EMC Corporation. All rights reserved.
• Heterogeneous nodes (different CPU models,
memory footprints, etc)
• Supports elastic scaling
– Nodes can be added for capacity
– Nodes can be removed for maintenance
CLUSTER NODES

19. 19© Copyright 2016 EMC Corporation. All rights reserved.
• Has been in production use for 3+ years
• 10k+ node scale
• Recognized by users and the community
• Tasks dispatching scale linearly over # nodes (2010)
STABILITY AND PERFORMANCE

20. 20© Copyright 2016 EMC Corporation. All rights reserved.
POWERED BY MESOS

21. 21© Copyright 2016 EMC Corporation. All rights reserved.
• Marathon
– manages long-running applications.
Provides both a UI and REST API
• Chronos
– is a replacement for cron to schedule batch
jobs and other workloads.
• Mesos-DNS
– enables DNS based service discovery
• Etc, etc, etc…
RICH APPLICATIONS AND ECOSYSTEM

22. 22© Copyright 2016 EMC Corporation. All rights reserved.
• Runs a wide variety of workloads
• On commodity hardware
• Efficient utilization of resources
• High Availability / Fault Tolerant
• UI and API management
WHAT WILL MESOS DO FOR ME?

23. 23© Copyright 2016 EMC Corporation. All rights reserved.
OR TRY TO BUILD ONE FROM SCRATCH

24. 24© Copyright 2016 EMC Corporation. All rights reserved. 24© Copyright 2016 EMC Corporation. All rights reserved.
WHO IS MESOSPHERE?

25. 25© Copyright 2016 EMC Corporation. All rights reserved.
MESOSPHERE: BORN OF THE HYPERSCALE ERA
Mesosphere is well funded
$50M by Tier 1 investors:
Andreessen Horowitz & Khosla Ventures
2009 2013 2014 2015
First Mesosphere DCOS
lighthouse customers
Tens of thousands of containers launched
Expanded Operations
San Francisco (2013)
Hamburg (2014)
New York (2015)
Berlin (2015)
Mesosphere Founded
Key engineering leaders from Twitter,
Airbnb - companies behind open-
source tech
Apache Mesos built at UC Berkeley
● Core technology from AMPlab
● Corporate large-scale production deployments
begin (e.g., Apple, Twitter, Salesforce)
Growing Ecosystem &
DCOS General Availability
Now a breeze to install modern app
services (e.g., Hadoop, Spark,
Cassandra)

26. 26© Copyright 2016 EMC Corporation. All rights reserved.
DELIVERING OUTCOMES FOR ENTERPRISES
DEVELOPER
AGILITY
HYPERSCALE
OPERATIONS
DATA
AGILITY
Highly Efficient & Scalable IT
Distributed Framework
Pooling
Container Orchestration
Cont. Integration/Delivery
Developer Cloud
Self-Service Container
Runtime
Big Data
Internet Of Things
Reactive applications

27. 27© Copyright 2016 EMC Corporation. All rights reserved.
THE DATACENTER IS THE NEW SERVER
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER
VIRTUAL
MACHINE
PARTITION
(LPAR)
UNIT OF
INTERACTION
● ERP, CRM,
PRODUCTIVITY, MAIL &
WEB SERVER
● LINUX, WINDOWS
● DATA /
TRANSACTION
PROCESSING
● UNIX, IBM OS/360
DEFINITIVE
APPS AND OS
● ERP, CRM,
PRODUCTIVITY, MAIL &
WEB SERVER
● HYPERVISOR + GUEST
OS
● BIG DATA, INTERNET
OF THINGS, MOBILE
APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR
DEVELOPING AND
RUNNING APPS
● BIG DATA, INTERNET
OF THINGS, MOBILE
APPS
● DATACENTER
OPERATING SYSTEM
(DCOS)

28. 28© Copyright 2016 EMC Corporation. All rights reserved.
MESOSPHERE DCOS ACCELERATES ENTERPRISES TO
A MODERN CLOUD STACK WITH THE DATACENTER FORM
Datacenter Operating System (DCOS)
Distributed Systems Kernel (Apache Mesos)
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern Cloud App Components
Distributed systems kernel to
abstract resources
Killer “apps” ecosystem
Cont Integration/Cont Delivery
Framework “app” architecture
to consistently run on top of
kernel
User Interface (GUI & CLI)
Core system services
(e.g., distributed init, cron,
service discovery, package mgt &
installer, storage)
Any Infrastructure (Physical, Virtual, Cloud)

29. 29© Copyright 2016 EMC Corporation. All rights reserved.
DCOS BRINGS HYPERSCALE TO ENTERPRISES
Desktop Apps (e.g., Safari,
Adobe Photoshop, Itunes)
GUI (Aqua)/CLI
Personal computer
- App store
- OpenGL
- Advanced UI gestures
BSD Unix
Mobile Apps (e.g., Spotify,
Evernote, WhatsApp)
Android GUI
Mobile devices
- Telephony Manager
- Battery management
- External storage support
Linux
Distributed systems
(e.g., Docker, Spark,
Hadoop, Cassandra)
GUI/CLI (DCOS CLI)
Full Datacenter / Cloud
- Container orchestration
- Distributed batch jobs
- Persistent storage mgmt
Apache MesosKernel
Applications
Form Factor
User Interface
OS Services
(highlights)
Computing for the masses Mobile apps for the
masses
Datacenter-scale apps for
mainstream enterprises
Benefits

30. 30© Copyright 2016 EMC Corporation. All rights reserved. 30© Copyright 2016 EMC Corporation. All rights reserved.
HOW EMC {CODE} FITS WITH MESOS

31. 31© Copyright 2016 EMC Corporation. All rights reserved.
• Containers are ephemeral therefore stateless
• No state? Means boring Application
• Container vendors started introducing state by way
of local disks
• Something better?
• EMC {code} brings storage expertise to the table
HOW EMC {CODE} FITS WITH MESOS?

32. 32© Copyright 2016 EMC Corporation. All rights reserved.
• Enable external storage
• Enhance frameworks to embrace and consume
external storage
• Run EMC storage platforms as scale out data
services
EMC {CODE} GOALS WITH MESOS

33. 33© Copyright 2016 EMC Corporation. All rights reserved.
• Persistent
– Containers can come and go,
but continue where they left off
– Application Data, Logs, Audit
Trail, etc
• External
– Storage lives outside the Mesos
Agent node
– High Availability
– Tolerates node failure
PERSISTENT EXTERNAL STORAGE

34. 34© Copyright 2016 EMC Corporation. All rights reserved.
• REX-Ray
– Provides a vendor agnostic storage orchestration engine
– AWS, GCE, ScaleIO, XtremIO, many many more
– Github: https://github.com/emccode/rexray
• mesos-module-dvdi
– Provides a hook to Mesos agent nodes to provision, attach,
and detach external storage
– Github: https://github.com/emccode/mesos-module-dvdi
HOW DO WE ACHIEVE THIS?

35. 35© Copyright 2016 EMC Corporation. All rights reserved.
• We can run “real world” applications
• Persistence – DBs
• Highly Available, Fault Tolerant
• Audit trail – Transaction Logs
• Not worry about the “2am call”
STATELESS TO STATEFUL

36. 36© Copyright 2016 EMC Corporation. All rights reserved.
THE 2AM CALL…

37. 37© Copyright 2016 EMC Corporation. All rights reserved. 37© Copyright 2016 EMC Corporation. All rights reserved.
DEMO

38. 38© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 1: TWEETER - ARCHITECTURE
Datacenter Operating System (DCOS)
Containerized
Microservice
Kafka Distributed
Messaging
system
Cassandra
Distributed
Datastore
Spark Data
Analytics JobsDistributed
Software Load
Balancers
“Tweeter” Service

39. 39© Copyright 2016 EMC Corporation. All rights reserved.
• Running on Amazon EC2
– Mesos
– Marathon
– Mesos DNS
• Production HA Cluster
– 3 x Master + Zookeeper Nodes (t2.micros)
– 4 x Slave Nodes with Docker Support (t2.medium)
DEMO 2: CONFIGURATION

40. 40© Copyright 2016 EMC Corporation. All rights reserved.
• Production Website (MVC)
– Web UI (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restclient
– REST API (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restserver
– PostgreSQL (Docker Container) backed by Persistent
External Storage
• Using REX-Ray
• Using mesos-module-dvdi
• Simple Web App - User Account Management
DEMO 2: PRODUCTION WEBSITE

41. 41© Copyright 2016 EMC Corporation. All rights reserved.
• Deploy Model, View, Controller using REST API
– Observe creation of each component
– Observe PostgreSQL DB attaching to EBS volumes
– Using browser, log into Account Management UI and
add/delete records
– Observe data from Web UI being added to PostgreSQL table
• Simulate Failure
– Kill EC2 Instance
– Watch task recreation and reattach to EBS volumes
DEMO 2: PRODUCTION WEBSITE

42. 42© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 2: INITIAL DEPLOY

43. 43© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 2: SCALE IT UP

44. 44© Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container World
Wednesday 3PM
Joshua Bernstein
VP of Technology for ETD
Tobi Knaup
CEO & Co-Founder of Mesosphere
Guru Session

45. 45© Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode
@strikesme
@dvonthenen
emccode.com
community.emccode.com
Come visit us at
Booth #1044
or in the vLab
Questions?

46. 46© Copyright 2016 EMC Corporation. All rights reserved. 46© Copyright 2016 EMC Corporation. All rights reserved.