Open collaboration in the Moby Project

Copyright©2017 NTT Corp. All Rights Reserved.
Akihiro Suda ( @_AkihiroSuda_ )
NTT Software Innovation Center
Open collaboration
in the Moby Project
Open Source Forum (Nov 15, 2017)
https://slideshare.net/AkihiroSuda

2
• Software Engineer at NTT
• github: @AkihiroSuda
• Twitter: @_AkihiroSuda_
• Docker Moby core maintainer (github.com/docker/docker
moby/moby)
• In April 2017, Docker [ as a project ] transited into Moby.
• Now Docker [ as a product ] has been developed as one of downstreams
of Moby.
: ≒ :
RHEL Fedora
Who am I

3
• BuildKit initial maintainer (github.com/moby/buildkit)
• Next-generation `docker build`
• Executes DAG vertices of Dockerfile-equivalent concurrently
• Soon: cache-aware distributed mode
• containerd maintainer (github.com/containerd/containerd)
• Industry-standard container runtime
• Can be used as a Docker-replacement for Kubernetes
Who am I

4
• What is the Moby Project
• Recent collaborative work in the Moby Project
• Governance of the Moby Project
• Case study: How I became a maintainer (and how you can!)
Agenda

5
What is the Moby Project?

6
Anyone interested in adding
"our Moby Project" to Wikipedia?

7
• A collaborative project that provides the "lego set" that
can be assembled into container-based systems
runc
BuildKit
Moby registry
DataKit
VPNKit
HyperKit
Moby engine
(dockerd)Moby tool
libnetwork
libentitlement
projects under http://mobyproject.org/projects/ as of Nov. 9.

8
• 9,149 Contributors
• 8,800 Pull Requests / Year
• Maintainers' affiliations:
https://www.slideshare.net/chanezon/dockercon-eu-2017-recap/51
Affiliations of the maintainers (including curators) of the projects under http://mobyproject.org/projects/ as of Nov. 9.
Maybe not 100% accurate :P
ADP
Atomic
Axway
Cloudflare
Codeship
CoreOS
Docker
Dropbox
Facebook
Google
Hamburger Software
Huawei
IBM
InfluxData
InfoSiftr
Infoblox
Kiratech
Microsoft
NTT
Rancher Labs
Red Hat
SUSE
Suranaree Univ. of Tech.
Tencent
vente-privee
Anyone is eligible!

9
• Docker has been developed collaboratively by Docker, Inc.
and contributors from various affiliations
• But Docker, Inc. has their own product roadmap
• In early 2017, Docker, Inc. and the community decided
splitting the Docker Project from the Docker Product, so as
to encourage more open collaboration
• Inspired by the Fedora community
• In April 2017, the Moby Project was established as the
successor of the Docker Project
Origin of the Moby Project

10
Relationship between Moby and its downstreams
Docker Community Edition Docker Enterprise Edition
Balena: Moby-based container engine for IoT, by Resin.io
+ Support, GUI..
Add your own downstream here...
?
Other downstreams
Docker, Inc.'s products
Similarity

11
• `dockerd` command (daemon) is part of Moby
• Will be renamed to `moby-engine` soon
• `docker` command (CLI) is NOT part of Moby
• Because UX is basically out of scope of Moby
• Solely maintained by Docker, Inc. But still opensource.
• Mac/Win and cloud installers are NOT part of Moby
• Docker, Inc. 's proprietary software at the moment
• Dockerfile is being removed from Moby... But no worry!
• Moby BuildKit provides a new low-level instruction set
Moby is not Docker
https://forums.mobyproject.org/t/topic-find-a-good-and-non-confusing-home-for-the-remaining-monolith/37/66
https://github.com/moby/buildkit/issues/163

12
• Announcement at DockerCon EU (October 17, 2017)
Recent collaborative work in the Moby Project
Kubernetes
Docker
containerd
Docker
Kubernetes
containerd
Transition
API translator (unreleased)
Container Runtime Interface (CRI)

13
• This was not a surprise, because Docker, Inc. and the
Moby community has been already collaboratively working
on Kubernetes-related stuff for a year
https://blog.mobyproject.org/moby-and-kubernetes-bf888ab31e38

14
• In addition to support for Kubernetes, the Moby community
is going to decouple more components
• People can work on their own respective stuff tactically,
while collaborating on the common Moby "lego set"
• Examples: containerd, LinuxKit, InfraKit, BuildKit..
Place for
open collaboration
Respective work

15
containerd: Industry-standard container runtime
• Simpler architecture than the "monolith" of Docker
• Set of decoupled subsystems
• More collaborative than the past Docker in the pre-Moby era
• Donated to CNCF
• Can be used as a Docker-replacement for Kubernetes (and so on)
• CRI-containerd, the glue module for k8s is mainly maintained by Google employees
(k8s incubator)
runc
containerd v1.0
CRI-containerd
Kubernetes
Transition
Kubernetes
Docker (Moby)
runc
containerd v0.2
Docker (Moby)
Docker 18.XX (unreleased)

16
containerd: Industry-standard container runtime
• containerd provides "a la carte" of well-decoupled
subsystems
• Developers of containerd-based system can chose only what they want
• They can add their own custom plugins, while collaborating on the
common plugins
OCI Image tarballDocker registry
OverlayFS btrfs
Prometheus
OCI Runtime
OCI Image tarball
OCI Runtime
Custom FS plugin
Place for
open collaboration
Home-made
container system
Assemble

17
• Assembles VM images for running certain service like
Docker / Kubernetes
• Usecases are not limited to Docker / Kubernetes
LinuxKit: Toolkit for custom Linux distribution
LinuxKit
Kubernetes
LinuxKit
Docker
Redis
LinuxKit
Hyperkit
EC2
Place for
open collaboration
Docker, Inc.'s
Docker for Mac
Home-made
"RedisOS"
Installer for mac

18
• Deploys distributed system like Docker Swarm-mode to IaaS like EC2
• Self-healing
• Recreate instances if the actual state differs from the desired state
• Now supports deploying Kubernetes as well
• Yuji Oshima (my colleague at NTT) implemented this with the community
• Through this collaboration, he became an InfraKit maintainer
InfraKit: Toolkit for infrastructure orchestration
Infrakit
LinuxKit LinuxKit
EC2 EC2
LinuxKit
EC2
Docker Swarm-mode / Kubernetes

19
• Provides DAG-based low-level build instructions (LLB)
• DAG: Directed Acyclic Graph
• DAG allows running LLB instructions concurrently
• Cache-aware distributed mode (on Kubernetes) is on plan
BuildKit: Next-generation `docker build`
image://alpine
Image
git://foo/bar
image://gcc
Run("apk add ..")Run("make")

20
• LLB is expected to be compiled from human-friendly
languages such as Dockerfile
• Dockerfile is going to be officially under Docker, Inc.'s control
• People can invent Dockerfile-alternative languages, while
collaborating on the common LLB spec and toolkit
BuildKit: Next-generation `docker build`
Compile
(Docker, Inc.'s) Dockerfile
BuildKit LLB
Other languages
Place for
open collaboration
Respective work
https://forums.mobyproject.org/t/topic-find-a-good-and-non-confusing-home-for-the-remaining-monolith/37/66
https://github.com/moby/buildkit/issues/163

21
Even useful for non-container
usecases, as a generic
concurrent / distributed build
toolkit
BuildKit: next-generation `docker build`
We need more usecase and
design feedback
from the community

22
• Previously, Solomon Hykes (CTO, Docker, Inc.) was the
BDFL
• Benevolent Dictator For Life
• On November 13, Technical Steering Committee (TSC) was
established as the replacement for the BDFL role
• Elected by maintainers
• A single company cannot hold more than 1/3 seats
• 7 members
• Docker x2, Codeship, IBM, InfoSiftr, SEAL Systems, vente-privee (2017-2019)
Moby governance

23
Moby Project, CNCF, and OCI
OCI
Moby Project
CNCFrunc
Image Spec
Runtime Spec
Kubernetes
Moby engine
...
... ...
• Some components under the Moby umbrella belong to
other organizations as well
• Moby TSC will help cross-project and cross-organization
collaboration
Overlapping projects under http://mobyproject.org/projects/ as of Nov. 9.

24
• Established in 2015 under the Linux Foundation umbrella
to create the vendor-neutral container standards
• by Docker, CoreOS, and others
• In July 2017, OCI Runtime Spec v1.0 & Image Spec v1.0
were announced
• Moby projects implement OCI specs; so they can be easily
integrated with other container projects
• runc (the reference OCI runtime) is governed by OCI, but
still considered to belong to the Moby Project
OCI: Open Containers Initiative
https://github.com/moby/tsc/issues/1

25
• Currently, image distribution spec is out of the scope of
OCI
• But many OCI folks are interested in bring the distribution
spec into the OCI scope
• https://github.com/cyphar/parcel
• https://github.com/xiekeyang/oci-discovery
• Docker Registry implementation (and spec?) is being
moved to under the Moby Project
• Likely to have some influence on OCI, but still unclear :P
OCI: Open Containers Initiative
https://github.com/moby/moby/issues/35115

26
• Established in 2015 under the Linux Foundation umbrella
as well for hosting Kubernetes (Now 14 projects)
• containerd and Notary are CNCF projects but still belong
to the Moby Project
• CNCF and k8s host some competing projects as well, but
they are collaboratively implementing the common spec
(OCI & k8s CRI: Container Runtime Interface)
• CRI-O (k8s incubator)
• rkt (CNCF) and rktlet (k8s incubator)
CNCF: Cloud Native Computing Foundation

27
Landscape 2018
Moby engine
containerd
CRI-containerd
Docker CLI
BuildKit
Kubernetes
CRI-O
runc
Dockerfile compiler New CLI?New lang?
?
Part of Moby? Proprietary?
Moby BuildKit LLB
OCI Image Spec
Docker Registry API (Moby? OCI?)
OCI Runtime Spec
k8s CRI
Moby Engine API

28
• Maintainers (aka "committers") can:
• Approve other contributors' pull requests (with 2 LGTMs usually)
• Manage GitHub issues
• Elect TSC members
• Add and remove other maintainers (with 66% approval vote)
• Maintainers are elected from active contributors who:
• Send pull requests (Bug fix, Enhancement, New feature...)
• Review other contributors pull requests
• Triage GitHub issues
Who are maintainers and how they are elected
Anyone is eligible!

29
• Began contribution to Docker in December, 2015
• Motivation: I was working on some fault injection tool (github.com/osrg/namazu), and
got stuck in "false bug" of Dockerized ZooKeeper due to AUFS hang-up, and needed to
look into the AUFS issue: #18180
• This is not a bug of Docker but mainly tracked in Docker community (AUFS maintainer fixed
that issue)
• Became a Docker maintainer in November, 2016
• Docker  Moby in April, 2017
• (I think) Mainly contributed to filesystem issues
• Both AUFS and overlayfs have some stability and compatibility issues
• Also proposed some new features
• `docker network prune` (merged): #27525
• introspection mount (procfs-like stuff for containers. unmerged yet): #24893, #26331
• TCP port forwarder (withdrawn): #26365
• ...
Case study: How I became a Docker/Moby maintainer

30
• Initial maintainer from the beginning of the project (2017
summer)
• I proposed DAG-based builder (but without idea of LLB)
• Coincidently, Tõnis Tiigi (Docker, Inc.) was planning
similar but even better idea, which turned into BuildKit
• Tõnis invited me to an initial maintainer of BuildKit 
Case study: How I became a BuildKit maintainer

33
• Began contribution in December, 2016
• Became a maintainer in September, 2017
• Mainly contributed to filesystem and image issues
• Though contribution to Docker / Moby, I found filesystem issues are
hard to maintain
• I needed to reform containerd interfaces and data formats for my
experimental OCI Image extension
• FILEgrain: transport-agnostic, fine-grained content-addressable container image
layout (github.com/AkihiroSuda/filegrain)
Case study: How I became a containerd maintainer

34
• Collaborativeness (the most important)
• "Best way to became an #opensource maintainer is to start helping like one" -
@icecrime
• Comprehensiveness
• Issue analysis, Bug-fix, enhancement, feature addition...
• But when you plan to add a big feature, please coordinate with maintainers in GitHub
issue or in Slack before opening a PR!
• Continuity
• One-shot contribution is always welcomed, but maintainership
requires continuous activity
• But no need to inflate the numbers of your git commits
And how you can become a maintainer! (my personal view)

35
• Moby engine
• Recently integrated containerd v1.0 runtime. We need to make sure
there is no regression.
• The next step is to integrate containerd v1.0 snapshot subsystem.
(Much harder for compatibility)
• BuildKit
• Docker compatibility is not stable. Testing is highly welcomed.
• Design for distributed mode is still under discussion.
• containerd
• Testing and performance optimization are welcomed.
• And more!
Good chances to contribution (my personal view)

36
• The Moby Project is getting more collaborative
• You can contribute and become a maintainer
Recap
https://blog.docker.com/2017/04/introducing-the-moby-project/

Open collaboration in the Moby Project

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Open collaboration in the Moby Project

Similar a Open collaboration in the Moby Project (20)

Más de Akihiro Suda

Más de Akihiro Suda (20)

Último

Último (20)

Open collaboration in the Moby Project