Mesos and Mesosphere are popular platforms for managing the consumption of data center resources and workloads. Recent enhancements to Mesos extend its management scope to go beyond resources supplied by individual cluster nodes. For example, Mesos can manage external storage from a platform such as ScaleIO. Stop by and learn about the data center of the future running Mesos.

1. © Copyright 2016 EMC Corporation. All rights reserved.© Copyright 2016 EMC Corporation. All rights reserved. 1
EXTENDING MESOS FOR STORAGE AS A GLOBAL RESOURCE
STEVE WONG
DEVELOPER ADVOCATE – EMC{code}
GREG MANN
SOFTWARE ENGINEER – Mesosphere

2. © Copyright 2016 EMC Corporation. All rights reserved. 2
GOT CONTAINERS AT SCALE?
YOU NEED A SCHEDULER…

3. © Copyright 2016 EMC Corporation. All rights reserved. 3
ASSUME 1000’S OF SERVICES
• How do you place an
assortment of
containers onto your
servers efficiently?

4. © Copyright 2016 EMC Corporation. All rights reserved. 4
WHAT’S A SCHEDULER?
• Manages compute nodes
– Takes reports from each node regarding their
health and available capacity
• Manages incoming work requests
– Accepts job requests
– Based on requirements, decides which node
should fulfill each request
– May attempt to optimize job placement to
maximize locality, utilization, etc.
on demand
scheduled
batch

5. © Copyright 2016 EMC Corporation. All rights reserved. 5
When you build a cluster, there’s no getting around it:
you are building a distributed system
• Difficult to design, build and operate
• Introduces complexity
Mesos takes on some of the "heavy lifting" of deploying and
operating a distributed system
MESOS AS A CLUSTER MANAGER

6. © Copyright 2016 EMC Corporation. All rights reserved. 6
A distributed systems kernel
• Abstracts many independent nodes into a single pool
• Negotiates among multiple schedulers
– Competing schedulers can suffer from resource starvation
– Mesos offers resources to frameworks to ensure fairness
WHAT IS MESOS?

7. © Copyright 2016 EMC Corporation. All rights reserved. 7
WHAT IS MESOS?
Marathon

8. © Copyright 2016 EMC Corporation. All rights reserved. 8
• Where it started
• Recent
enhancements
• Roadmap
MESOS AND STORAGE

9. © Copyright 2016 EMC Corporation. All rights reserved. 9
CLUSTER NODES OFFER LOCAL RESOURCES

10. © Copyright 2016 EMC Corporation. All rights reserved. 10
MESOS MASTER PRESENTS OFFERS TO
FRAMEWORKS
Marathon

11. © Copyright 2016 EMC Corporation. All rights reserved. 11
• Where it started
• Recent
enhancements
• Roadmap
MESOS AND STORAGE

12. © Copyright 2016 EMC Corporation. All rights reserved. 12
RESERVE CLUSTER NODE HOSTED VOLUME

13. © Copyright 2016 EMC Corporation. All rights reserved. 13
External volume
mounts were added
in 2015
• Provides
Availability + Scale
MOUNT A VOLUME FROM EXTERNAL STORAGE

14. © Copyright 2016 EMC Corporation. All rights reserved. 14
• Where it started
• Recent
enhancements
• Roadmap
MESOS AND STORAGE

15. © Copyright 2016 EMC Corporation. All rights reserved. 15
NOT ALL RESOURCES ARE LOCAL

16. © Copyright 2016 EMC Corporation. All rights reserved. 16
AGENT FAILURE
• Agents are the
source of truth for
resource
information

17. © Copyright 2016 EMC Corporation. All rights reserved. 17
AGENT FAILURE
• When an agent fails
this information is
lost, but so are the
resources!

18. © Copyright 2016 EMC Corporation. All rights reserved. 18
AGENT FAILURE
• When a failed
agent was using a
global resource,
the resource is still
available!

19. © Copyright 2016 EMC Corporation. All rights reserved. 19
AGENT FAILURE
• The state of this
resource must be
stored off the
agent

20. © Copyright 2016 EMC Corporation. All rights reserved. 20
NEED A NEW ABSTRACTION
• We need a third
entity to track the
use of global
resources

21. © Copyright 2016 EMC Corporation. All rights reserved. 21
• Option #1:
Integrate global
resources into the
Mesos offer cycle
HOW TO INTEGRATE GLOBAL RESOURCES?

22. © Copyright 2016 EMC Corporation. All rights reserved. 22
• Option #2: Handle
global resources
“out-of-band”
HOW TO INTEGRATE GLOBAL RESOURCES?

23. © Copyright 2016 EMC Corporation. All rights reserved. 23
Mesos was originally designed to manage compute
node resources
• Storage in this context meant DAS
A JIRA proposal exists to include global resources in
the Mesos offer model, but this may not be necessary.
• Feedback is welcome!
HOW TO INTEGRATE GLOBAL RESOURCES?

24. © Copyright 2016 EMC Corporation. All rights reserved. 24
HOW DO YOU TRACK WHAT STORAGE YOU
HAVE?
… AND WHERE IT’S IN USE?

25. © Copyright 2016 EMC Corporation. All rights reserved.
Create
Utilize
Retire
MESOS INTERFACES WITH STORAGE
Lifecycle management with
governance
Mount managementAssignment to applications

26. © Copyright 2016 EMC Corporation. All rights reserved. 26
A LARGE SCALE INSTALLATION NEEDS TO
ALLOW FOR MULTIPLE STORAGE PROVIDERS
What does this remind
you of?

27. © Copyright 2016 EMC Corporation. All rights reserved. 27
MESOS = 2 LEVEL CONTAINER SCHEDULER

28. © Copyright 2016 EMC Corporation. All rights reserved. 28
TWO LAYER EXTERNAL STORAGE SCHEDULING

29. © Copyright 2016 EMC Corporation. All rights reserved. 29
STORAGE SCHEDULER = DELIVER SCALE OUT
Scale out is not just about adding capacity.
A Storage Scheduler will let you:
• Add, remove, replace storage provider platforms
– in or out of public cloud
• Add, remove, replace schedulers
• This is about giving you flexibility and
shifting the burden of keeping up to vendors

30. © Copyright 2016 EMC Corporation. All rights reserved. 30
POLLY AS A MESOS FRAMEWORK

31. © Copyright 2016 EMC Corporation. All rights reserved. 31
DCOS: THE DATACENTER OPERATING SYSTEM
learn more at mesosphere.com

32. © Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container World
Wednesday 3PM
Joshua Bernstein
VP of Technology for ETD
Tobi Knaup
CEO & Co-Founder of Mesosphere
Guru Session

33. © Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode
@cantbewong
mesosphere.com
emccode.com
community.emccode.com
Come visit us at
Booth #1044
or in the vLab
Questions?

34. © Copyright 2016 EMC Corporation. All rights reserved.© Copyright 2016 EMC Corporation. All rights reserved. 34