Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1FjjXpZ. Andrew Kennedy talks about the reasons for creating a Docker cloud and how they realized that to do this properly they needed first class networking to handle composite distributed applications such as Riak. It was a short step from this to using Brooklyn itself to bootstrap a Docker cloud effectively colonizing the infrastructure. And so Clocker was born. Filmed at qconlondon.com. Andrew Kennedy is a Senior Software Engineer at Cloudsoft and the founder of the Clocker project. He is a contributor to several Open Source projects including Apache jclouds and Apache Qpid and is also a founder member of the Apache Brooklyn project.

1. QCon
London
2015;
Docker
Track
Andrew
Kennedy
@grkvlt
Clocker
How
to
Train
Your
Docker
Cloud

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/docker-cloud-clocker

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. @grkvlt
Introduc/on
• Andrew
Kennedy
– Clocker
Project
Founder
and
Lead
Engineer
– Open
Source
and
Distributed
Systems
– github.com/grkvlt
• CloudsoG
Corpora/on
– Open
Source
Applica/on
Management
Specialists
– We’re
Hiring…

5. @grkvlt
Agenda
• Clocker
Introduc/on
• What
is
a
Docker
Cloud?
• Training
a
Docker
Cloud
• Demonstra/on
• Roadmap

6. Clocker
IntroducBon

8. @grkvlt
What
does
it
do?
1. Spins
up
and
Manages
Docker
Clouds
2. Serves
up
Containers
on
Demand
3. Manages
Composite
Applica/on
Deployments
on
Docker

9. @grkvlt
What
does
it
provide?
• Mul/
Host
and
Mul/
Container
Applica/ons
• Seamless
Networking
– Communica/on
Between
Services
• Orchestra/on
and
Clustering
– Control
of
Containers
– Container
Management

10. @grkvlt
Who
is
using
it?
• Tes/ng
and
Proof
of
Concept
Stage
– Financial
Services
– Insurance
• Produc/on
– Mul/-­‐tenant
Applica/on
Trial
– Container
per
service
– Via
IBM
BlueMix

11. @grkvlt
Where
can
I
find
it?
• Open
Source
• Apache
2.0
Licensed
• h[p://clocker.io
• Status
• 0.7.0
Beta
Release
• 0.8.0-­‐SNAPSHOT
Development

12. What
is
a
Docker
Cloud?

13. @grkvlt
Clocker
and
Brooklyn
• What
is
it?
– Brooklyn
Applica/on
– Brooklyn
Loca/on
• What
does
it
provide?
– First
Class
Docker
Support
in
Brooklyn
– Op/mized
Brooklyn
Blueprints
for
Docker

14. Apache
Brooklyn
• Apache
Brooklyn
– Open
Source
– Java
– Donated
to
the
Apache
SoGware
Founda/on
– ASF
Incubator
Status

15. @grkvlt
Apache
Brooklyn
• Applica/on
Management
Pla_orm
• Deploy,
Manage
and
Monitor
Blueprints
• Provisioning,
Installa/on
and
Customiza/on
• Management
– AutoScaling,
Resilience,
Performance,
Security

16. Apache
jclouds
• Java
Cloud
Library
• API
Agnos/c
– CloudStack,
OpenStack,
AWS
EC2,
GCE…
• Create
Virtual
Machines
– Return
SSH
Endpoint
– Manage
Proper/es

17. @grkvlt
Apache
jclouds
• Drivers
for
REST
APIs
• Docker
Driver
– Wri[en
by
@turlinux
• Virtual
Container
– Using
SSH
Daemon
– Same
Endpoint
Type
as
VM
– Composi/on
on
any
Image
or
Dockerfile

18. @grkvlt
Docker
• Popular
– Large
Ecosystem
– Growing
– Complex
• Containers
– Isola/on
– Performance
– Composable

19. @grkvlt
Docker

20. @grkvlt
Brooklyn
Blueprints
• Describe
Applica/ons
• OASIS
CAMP
Standard
• List
of
Services
• Tree
Structure
• Sensors,
Effectors
and
Policies

21. @grkvlt
Blueprint
Example
name:
appserver-­‐w-­‐policy
services:
-­‐
type:
brooklyn.entity.webapp.ControlledDynamicWebAppCluster
initialSize:
1
memberSpec:
$brooklyn:entitySpec:
type:
brooklyn.entity.webapp.jboss.JBoss7Server
brooklyn.config:
wars.root:
http://search.maven.org/remotecontent?filepath=io/brooklyn/example/brooklyn-­‐example-­‐hello-­‐world-­‐sql-­‐webapp/0.6.0/brooklyn-­‐example-­‐hello-­‐world-­‐sql-­‐
webapp-­‐0.6.0.war
http.port:
8080+
java.sysprops:
brooklyn.example.db.url:
$brooklyn:formatString("jdbc:%s%s?user=%s&password=%s",
component("db").attributeWhenReady("datastore.url"),
"visitors",
"brooklyn",
"br00k11n")
brooklyn.policies:
-­‐
policyType:
brooklyn.policy.autoscaling.AutoScalerPolicy
brooklyn.config:
metric:
$brooklyn:sensor("brooklyn.entity.webapp.DynamicWebAppCluster",
"webapp.reqs.perSec.windowed.perNode")
metricLowerBound:
10
metricUpperBound:
100
minPoolSize:
1
maxPoolSize:
5
-­‐
type:
brooklyn.entity.database.mysql.MySqlNode
id:
db
name:
DB
HelloWorld
Visitors
brooklyn.config:
datastore.creation.script.url:
https://github.com/apache/incubator-­‐brooklyn/raw/master/usage/launcher/src/test/resources/visitors-­‐creation-­‐script.sql

22. @grkvlt
Applica/on
Management
• Policies
• Sensor
Data
Driven
• Auto
Scaling
• Cluster
Management
• Resilience
and
Failover
• Follow
the
Sun

23. @grkvlt
Docker
Cloud
• Brooklyn
Blueprint
for
Docker
Cluster
• Docker
Engine
on
Cloud
VM
– Configura/on
for
TLS
Cer/ficates
– Setup
Volumes
(Logging?)
– Install
Agents
(Security?
SDN?)

24. @grkvlt
Docker
Cloud
1. On-­‐demand
2. Mul/-­‐Tenant
3. Hardware
Independent
4. Applica/on
Level

25. @grkvlt
Architecture

26. @grkvlt
Clocker
Features
• Applica/on
Deployment
– Oasis
CAMP
YAML
Blueprint
– TOSCA
in
Development
– Same
as
Core
Brooklyn
• Mixed
Des/na/ons
– Some
Virtual
Machines
– Some
Bare
Metal
– Some
Containers

27. @grkvlt
Clocker
Features
• Docker
Extensions
to
Brooklyn
– Dockerfile
or
Image
Specifica/on
for
Installa/on
– Placement
Strategies
for
Containers
– Create
Docker
Images
and
Networks
• Manages
Docker
Engine
– Deployment
and
Management
– Installa/on
and
Configura/on
– SoGware-­‐Defined
Networking

28. @grkvlt
Clocker
Orchestra/on
Docker
Engine
Virtual
Machine
Container
Clocker
Network
Segment
SDN
Cloud

29. Training
a
Docker
Cloud

30. @grkvlt
Training…?
• Container
Management
• Placement
and
Provisioning
• Autonomics
– Policies
and
Scaling
• Headroom
• Networks

31. @grkvlt
Container
Management
• Sources
– Docker
Image
Defini/on
– Docker
Hub
– Dockerfile
– Brooklyn
En/ty
Defini/on
• Create
Image
Automa/cally
– Commit
or
Push
for
Reuse

32. @grkvlt
Container
Management
• Installa/on
of
Services
– Defined
by
Brooklyn
or
Dockerfile
– Common
to
all
En/ty
Instances
• Commit
Image
– Available
for
next
En/ty
• Push
Image
– Available
for
all
Hosts

33. @grkvlt
Container
Management
id:
dockerfile-­‐mysql
name:
"Docker
Hub
MySQL
Database"
origin:
"https://registry.hub.docker.com/_/mysql/"
locations:
-­‐
my-­‐docker-­‐cloud
services:
-­‐
type:
brooklyn.entity.container.docker.application.DockerfileApplication
id:
mysql
name:
"MySQL"
brooklyn.config:
docker.dockerfile.url:
"https://s3-­‐eu-­‐west-­‐1.amazonaws.com/brooklyn-­‐clocker/mysql-­‐5.6.tgz"
docker.container.environment:
MYSQL_ROOT_PASSWORD:
"s3cr3t”

34. @grkvlt
Container
Management
id:
dockerfile-­‐mysql
name:
"Docker
Hub
LAMP
Stack"
locations:
-­‐
my-­‐docker-­‐cloud
services:
-­‐
type:
docker:mysql:5.7.5
id:
mysql
env:
MYSQL_ROOT_PASSWORD:
"s3cr3t"
-­‐
type:
docker:grkvlt/myapp:latest
id:
application
env:
MYSQL_HOST:
component(”mysql").attributeWhenReady("host.hostname")

35. @grkvlt
Container
Placement
• Demand
– Adding
an
Applica/on
– Scaling
exis/ng
Applica/on
• Requirements
– Host
Loca/on
– Service
Resources
– CPU,
Memory

36. @grkvlt
Container
Placement
• Supply
– Choose
a
Host
from
available
– Create
new
Host
• Start
Container
– Set
CPU
and
Memory
– A[ach
to
Network

37. @grkvlt
Container
Placement
• Placement
Strategies
– Filter
Hosts
– Sort
Hosts
• Informa/on
from
– Docker
– Underlying
Machine
– Cloud
Environment
– En/ty
or
Service

38. @grkvlt
Container
Placement
• Placement
Strategies
– Random,
Depth
or
Breadth
First
– CPU
or
Memory
Usage
– Memory,
CPU
or
Container
Limits
– Geographic
Constraints
• User
Defined
– Java
Predicate

39. @grkvlt
Placement
Strategy
• Determinis/c
• Simple
– Predicate
and
Comparator
docker.container.strategies:
-­‐
$brooklyn:object:
type:
"brooklyn.location.docker.strategy.MaxContainersPlacementStrategy”
brooklyn.config:
maxContainers:
16
-­‐
$brooklyn:object:
type:
"brooklyn.location.docker.strategy.CpuUsagePlacementStrategy”
brooklyn.config:
maxCpu:
0.75

40. @grkvlt
Container
Provisioning
• Provisioning
Strategy
– New
Docker
Host
Loca/on
• Constraints
– Docker
Infrastructure
Constraints
– En/ty
or
Applica/on
Constraints
• User
Defined
Strategies
• Intelligent
Container
Orchestra/on

41. @grkvlt
Autonomics
• Brooklyn
Policies
• A[ached
to
En//es
in
Applica/on
– Nothing
Docker
Specific
• Elas/c
Scaling
– Cluster
Resizing
– Sensor
Driven

42. @grkvlt
Applica/on
Resilience
• Service
Resilience
and
Replacement
– Restart
Service
and
Container
– Applica/on
Level,
Not
Infrastructure
– Same
as
Cloud
• Could
be
Integrated
Further
– Snapshot
Running
Container
for
Restart

43. @grkvlt
Headroom
• Ensure
resources
available
• Based
on
MaxContainers
strategy
limit
– Or
Percentage
U/lisa/on
– Or
CPU
and
RAM
alloca/on
• Scale
Docker
Host
Cluster
Automa/cally
– Add
new
Docker
hosts
– Remove
empty
Docker
hosts

44. @grkvlt
SoGware-­‐Defined
Networking
• Needed
for
Seamless
Provisioning
• Host
to
Host
Communica/on
– Same
LAN
Segment
– No
Port
Forwarding
– Natural
Applica/on
Configura/on
• Ini/al
Driver
was
EPMD
Applica/ons

45. @grkvlt
Networking
Providers
• Standardized
Interfaces
• Pluggable
Providers
– Weave
– Metaswitch
Calico
– IBM
OpenDOVE
– Others?

46. @grkvlt
Networking
Providers
• Implementa/on
Agnos/c
– L2
over
L3
etc.
– Similar
to
Hypervisor
in
Clouds
• Generic
Interfaces
– Host
Component
– Service
Component
(or
Endpoint)

47. @grkvlt
Networking
Capabili/es
• Provide
Mul/ple
Networks
– Single
Applica/on
or
Shared
– Private
Addresses
– Segmented
by
CIDR
• Docker
Port
Forwarding
Access
– Debug
Mechanism

48. @grkvlt
Clocker
Networking
Container
Host
SDN
Bridge
Container
Internet
SDN
Gateway

49. @grkvlt
Networking
Capabili/es
• IP
Pool
Controlled
by
Clocker
• Clocker
Controls
Subnet
Alloca/on
– Applica/ons
Segmented
by
CIDR
• Delegate
to
SDN
or
Cloud
– Bring
your
own
IP
• Docker
Port
Forwarding
Access

50. @grkvlt
Future
Capabili/es
• Adding
More
SDN
Providers
• Name
Resolu/on
– BIND
and
DNSmasq
or
Weave
DNS
– Needed
for
JMX
et
al
– Enables
Many
More
En//es
• Adding
More
Supported
Blueprints

51. @grkvlt
Virtual
Network
En/ty
• New
Clocker
En/ty
• Provisions
a
Virtual
Network
• Looks
for
NetworkProvisioningExtension
– Registered
in
DockerLoca/on
by
SDN
Provider
– Could
also
add
to
JcloudsLoca/on
for
Clouds
• Part
of
Applica/on
Blueprint

52. @grkvlt
Network
Provisioning
• Create
Named
Virtual
Networks
• Uses
Available
Provider
– Currently
Weave,
Calico
and
IBM
Supported
-­‐
type:
brooklyn.networking.VirtualNetwork
networkId:
my-­‐application
cidr:
192.168.12.0/24
gateway:
192.168.12.1
dnsServers:
-­‐
8.8.8.8
brooklyn.config:
sdn.example.securityGroup:
"my-­‐security-­‐group"

53. @grkvlt
Network
Provisioning
• Allow
Minimal
(Zero!)
Configura/on
• Use
Sensible
Defaults
– Also
SDN
Specific
Configura/on
• Allocate
Address
Space
on
Demand
-­‐
type:
brooklyn.networking.VirtualNetwork
networkId:
database-­‐net
cidr:
192.168.34.0/24
-­‐
type:
brooklyn.networking.VirtualNetwork
networkId:
couchbase-­‐net

54. @grkvlt
Network
Provisioning
• A[ach
Containers
to
Networks
brooklyn.config:
network.list:
-­‐
couchbase-­‐net
-­‐
management-­‐net
• Create
Networks
as
Required
• Also
A[ach
to
VMs
and
Metal
– Where
Supported

55. DemonstraBon

60. @grkvlt
Features
• Orchestrated
Docker
1.5.0
deployment
with
Weave
SDN
integra/on
• Automated
a[achment
of
containers
to
mul/ple
dynamic
networks
• Brooklyn
applica/on
blueprints
with
network
topology

61. Clocker
Roadmap

62. @grkvlt
Roadmap
• Latest
Docker
1.5.0
Feature
Support
– Stats
command,
Read-­‐only
containers,
Named
Dockerfiles
and
more
• Swarm
– Add
Clocker
as
an
extension
to
Swarm
itself?
– Or
add
features
to
exis/ng
jclouds
driver?
• Compose
– Support
as
alterna/ve
blueprin/ng
language

63. @grkvlt
Roadmap
• Networking
– DNS
and
DNSmasq
Integra/on
– Rancher
Integra/on
– Open
vSwitch
Integra/on
• Gemng
Started
– Clocker
Dockerfile

64. @grkvlt
Roadmap
• Repositories
– Docker
Enterprise
Hub
– Bintray,
Ar/factory,
Quay.io
– Private
Repositories
• Applica/on
Defini/on
– TOSCA
– Docker
Compose

65. @grkvlt
Roadmap
• Plugin
API
will
be
helpful
• Also
projects
like
Powerstrip
• Provide
services
to
Docker
this
way
• Orchestra/on
• Policies
• Consume
wider
range
of
services
• Networking
and
Storage

66. @grkvlt
Summary
• Clocker
– Brooklyn
+
jclouds
+
Docker
+
Weave
+
…
– @clockercentral
• Solves
– Composite
Applica/on
Management
– Docker
Cloud
Networking
– Container
Placement
and
Provisioning

67. QuesBons?

68. @grkvlt
Web
Resources
http://clocker.io/
http://brooklyn.io/
http://docker.io/
http://github.com/zettio/weave/
http://abstractvisitorpattern.co.uk/
@clockercentral
@brooklyncentral
@cloudsoft

69. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/docker-
cloud-clocker