OpenStack images that fit your imagination - deep dive into container images in Kolla (OpenStack Summit 2017 in Sydney)

1. OpenStack images that fit your imagination -
deep dive into container images in Kolla
OpenStack Summit, Sydney, Nov 8 2017
Rich Wellum, Software Developer, Cloud Architecture, Lenovo, rwellum
Vikram Hosakote, Cloud Software Engineer, Cisco Systems, vhosakot

2. Agenda
• OpenStack Kolla
• What OpenStack Kolla solves
• Benefits of containers in OpenStack
• Architecture of Kolla
• How to use Kolla
• Deploy OpenStack in minutes – do not build
• How to customize Kolla images
• Demo

3. OpenStack Kolla
• Deploys OpenStack services in Docker containers
• Two ways to deploy:
• Ansible
• Kubernetes
• Three sub-projects:
• Kolla
• Kolla-ansible
• Kolla-kubernetes

4. What OpenStack Kolla solves
• Deploy OpenStack binaries or build from source
• Faster deployment
• Easy maintenance, re-configuration, patching and upgrades
• Containerized OpenStack services in registry
• Only one tool to do multiple things
• Pin:
• OpenStack version of each service
• Configuration files for each service

5. Which projects are users most interested in adopting in the future? Which projects are used by OpenStack deployments?
OpenStack User Survey (April 2017) - https://www.openstack.org/assets/survey/April2017SurveyReport.pdf

6. Kolla is easy to use
• DevStack
• Juju
• PackStack
• Fuel
• OpenStack-ansible
• TripleO
• Openstack-puppet
• OpenStack-chef

7. Benefits of containers in OpenStack
• Reproduce golden state easily – maintain, version, tag, upgrade
• No more “Works in DevStack” ;)
• Production-ready images
• Easy to override/highly flexible:
• Dockerfiles
• Jinja templates (configuration files)
• Dev mode (do Dev and Test quickly using Kolla)
• Portable, tested, replicated images in secure registries – DockerHub,
Quay, etc

8. Architecture of Kolla

9. Registry
(DockerHub)
GitHub
(kolla, kolla-
ansible, kolla-
kubernetes)
Deploy node
(Docker, Ansible,
Docker python,
Jinja2)
Compute
node
Compute
node
Compute
node
(Docker, OpenStack
services) Compute
node
Compute
node
Controller
node
(Docker, OpenStack
services) Compute
node
Compute
node
Storage
node
(Docker, OpenStack
services)
Management/admin network
API/service network
Storage/data network
OOB network
images, layersDockerfiles, ansible playbooks,
jinja templates (config files),
helm charts, k8s jobs
Target
nodes
deploy, re-configure,
upgrade
deploy, re-configure,
upgradedeploy, re-configure,
upgrade

12. Not just for vanilla OpenStack with default configs
• Kolla can be used for advanced configurations like
• Jumbo frames
• SR-IOV (PCI passthrough)
• Massive scale (hundreds of nodes)
• New networking technologies - OpenDaylight, OVS-DPDK, VPP*, OVN, NFV,
Tacker, Service Function Chaining (SFC)
• Supports plugins
• Ceph storage
• EFK (Elasticsearch, Fluentd, Kibana)
• Prometheus – cloud-native monitoring and alerting
• Customizations and template overrides

13. How to use Kolla
• Install:
• Install Ansible, Docker, Docker python and Jinaj2 on the deploy node
• Install Docker and Docker python on all the target nodes (bootstrap-servers.yml)
• Clone the kolla repos
• Build:
• Create /etc/kolla/kolla-build.conf (optional)
• kolla-build (lots of build options)
• Deploy kolla-ansible:
• Create Ansible inventory file and update globals.yaml
• kolla-ansible prechecks -i <inventory file>
• kolla-ansible deploy -i <inventory file>

14. Deploy OpenStack in minutes – do not build
• Pull pre-built golden images from DockerHub
(https://hub.docker.com/u/kolla/) instead of building images

15. Deploy OpenStack in minutes – do not build
• Kolla images are tagged per-release on DockerHub
• kolla-ansible –i inventory file pull

16. Building and Orchestrating Custom Kolla
Images and Containers(On Kubernetes!)
● Official Kolla Image building documentation:
https://docs.openstack.org/kolla/latest/admin/image-building.html
● These slides: http://bit.ly/2zl7WFF
● Orchestration tool used in this demo:
○ https://github.com/RichWellum/k8s.git
○ https://review.openstack.org/#/c/487972/
● Other orchestration options:
○ Ansible Playbook: https://review.openstack.org/#/c/457317/
○ Deployment Guide: https://docs.openstack.org/kolla-
kubernetes/latest/deployment-guide.html

17. Reasons why you might build custom Kolla
Images?1. Case example 1; your company builds and produces a proprietary OpenStack. Some services
are touched, some are not. For example Horizon has plugins and addons that uniquely identify
your company's look and feel, but changes to many other Services are not required.
○ Kolla provides tools to build Docker images containing an OpenStack Service. And a single
image for a Service up to the entire stack can be built.
2. Case example 2; as an OpenStack contributor you work with a Service - say Ironic. The bulk of
your work is done in devstack, a simulator. How cool would it be to develop in an OpenStack
environment running with production level code?
○ Kolla sub-projects: Kolla-Ansible and Kolla-Kubernetes provide the orchestration to run
containerized OpenStack.
3. Case example 3; Your company develops drivers unique to their hardware. Most of OpenStack
should just work, be easy to build and orchestrate in a real environment.
○ With Kolla the hard-work per Service has already been done; facilitating a low cognitive
overhead.

18. Building and Orchestrating
DEVELOPMENT
Local Source Code
ORCHESTRATION and OPERATIONS
Kolla-Kubernetes
Neutron Nova
Cinder
GlanceHeat
Ironic
Keystone Horizon
KOLLA BUILD
Tools to build
images
src/Keystone
+ LDAP or
SAML
src/Horizon
+ Custom
logo/skins
Reconfiguration
src/<Service>
Keystone Image
Horizon Image
<Service> Image
Kube
Schedul
er
Kube
Controll
er
Kube
SDN
Kube
DNS
Kube
etcd
Kube
Proxy
Tiller Kube
API
src code changed: Helm
delete / Helm install

19. Demo
● Start with source code for Horizon, make a customization change.
● Install Kolla.
● Compile a Horizon docker image with Kolla Build tools and store locally.
● Bring up Kubernetes Cluster and verify it is operational.
● Orchestrate a combination of UpStream(Ocata) Kolla containers from
Dockerhub and our new Horizon container on the Kubernetes Cluster.
● View the results on the Horizon GUI.
● Demo Video: https://www.youtube.com/watch?v=jyjQ_qy7wPU
● Demo Notes with all the steps are in the slides below for referenced.

20. DEMO

21. Wrap up - how to get involved
● Need a feature? Find a bug? Let us know!
● Gerrit workflow -
http://docs.openstack.org/infra/manual/developers.html
● Kolla IRC channel - #openstack-kolla
● File bugs, blueprints, track releases on Launchpad -
https://launchpad.net/kolla
● Attend weekly meetings -
https://wiki.openstack.org/wiki/Meetings/Kolla
● Contribute code - https://github.com/openstack/kolla
● Mailing list - openstack-dev@lists.openstack.org [kolla]

22. Demo Notes

23. Start with pulling your source code
● For the purpose of the rest of this presentation - the assumption is that
you are working from a VM…
● We are going to build custom Kolla Images for three services, Keystone,
Horizon and Cinder.
● Grab the custom OpenStack code - for example your company's source
code for keystone, horizon and cinder may reside on a git server
somewhere - clone it to a work-space, E.g:
○ git clone git@10.1.2.3:myopenstack/keystone.git
○ git clone git@10.1.2.3::myopenstack/horizon.git
○ git clone git@10.1.2.3::myopenstack/cinder.git

24. Grab Kolla Repository
● Grab the kolla repo which will give you access to the tools to build
images
○ git clone http://github.com/openstack/kolla
● Compile Kolla
○ sudo pip install -r ./kolla/requirements.txt
○ sudo -H pip install -U kolla

25. Generate files
● Generate a kolla-build.conf
○ sudo -H pip install tox
○ cd kolla; sudo -H tox -e genconfig
■ Note: (places build-conf in /etc/kolla !):
● Edit kolla-build.conf and add in source
○ vi etc/kolla/kolla-build.conf
○ install_type = source
■ But this is optional as it can be supplied as a parameter to kolla build
■ Note binary means that OpenStack will be installed from apt/yum. And the
source means that OpenStack will be installed from source code.

26. Edit kolla-build.conf and add pointer to source
● Edit kolla-build.conf and add in source pointers
○ vi etc/kolla/kolla-build.conf ⇐ note local etc not /etc
○ Lets add references to keystone, horizon and cinder source code
○ Note the reference - what will the rest of your OpenStack be running?
○ Type can also be git or url.
[keystone-base]
+ type = local
+ location = /home/rwellum/keystone/
+ reference = stable/ocata
[horizon]
+ type = local
+ location = /home/rwellum/horizon/
+ reference = stable/ocata
[cinder-base]
+ type = local
+ location = /home/rwellum/cinder/
+ reference = stable/ocata

27. Create a Docker Registry if needed
● For some scenarios (multi-node, sharing etc) it’s helpful to create a docker registry to store these images, E.g:
● kolla/tools/start-registry
● Or:
docker run -d
--name registry
--restart=always
-p 4000:5000
-v registry:/var/lib/registry
registry:

28. Now we are ready to build images (1/2)...
● You can choose to build the images locally (Like an All In One (AIO) Dev environment) or push to a
Docker Registry - if you intend to share these images or running multi-node.
● Change the base distro with the -b option:
○ kolla-build -b ubuntu
● Local build (4 different ways - why not?)
○ sudo kolla-build -t source horizon cinder keystone --config-dir
./kolla/etc/kolla/
○ Or: sudo which klollkolla-build -t source horizon cinder keystone --
config-file ./kolla/etc/kolla/kolla-build.conf
○ Or: copy kolla-build.conf to /etc/kolla (from ./kolla/etc/kolla/kolla-
build.conf
○ Or: sudo kolla/tools/build.py -t source --config-file
./kolla/etc/kolla/kolla-build.conf

29. Now we are ready to build images (2/2)...
● Registry multi-node
○ Add: “ --registry 172.22.2.81:5000 --push”
○ E.g. sudo kolla-build --registry 172.22.2.81:5000 --push -t source
horizon cinder keystone
○ To push images to a Dockerhub repository named mykollarepo:
○ sudo kolla-build -n mykollarepo --push -t source horizon cinder
keystone
● Add in build logs
○ Any issues in building this is where you start
○ Add: “--logs-dir horizon-log --debug“
○ E.g. sudo kolla-build --registry 172.22.2.81:5000 --push -t source
horizon cinder keystone --logs-dir horizon-log --debug

30. Lets check our images built...
● Lets check for valid generated images
rwellum@ubuntuk8s:~$ sudo docker images | grep horizon
kolla/centos-source-horizon 4.0.2 5182810deebb 2 minutes ago
1.077 GB
● The Tag ‘4.0.2’ is very important in Kolla…
● We’ll use this and the location of the image to orchestrate

31. How do we Orchestrate our new images (1/2)?
● So we have our custom images. Now we’d like to Orchestrate…
● Kolla is the main project - the tools to build OpenStack Images
● Kolla-Ansible is a sub-project to orchestrate with Kolla images
○ Industry proven, widely used, very reliable!
● Kolla-Kubernetes is another sub project to orchestrate with Kolla
images
○ Dev project, to run on Kubernetes cluster, new, exciting, be careful!
● This example is based on Kolla-Kubernetes and using a tool I wrote to
stand up an AIO kolla-kubernetes cluster…
● But it applies to any method, manual (following the bare-metal guide:
https://docs.openstack.org/kolla-kubernetes/latest/deployment-
guide.html) even Kolla-Ansible

32. How do we Orchestrate our new images (2/2)?
● Kolla-Kubernetes relies on a ‘cloud.yaml’ file to stand up
OpenStack
● In our example we want to run with our custom keystone,
horizon and cinder images
● All other Services will be pulled from dockerhub
● Grab k8s.py - deployment / orchestration tool:
○ git clone https://github.com/RichWellum/k8s.git
○ Replace with hopefully merged version

33. Orchestration continued...
● Standup OpenStack
○ Create a working directory - like ‘os’
○ Run k0:
■ ../k8s/ko.py eth0 eth1 --image-version ocata --edit-config
○ Note two interfaces required as per normal (MGMT and Neutron)
○ Note edit-config option
○ --help will show all the options available.
● Ko.py will proceed to install Kubernetes and start to bring up
OpenStack
● It will pause at the appropriate point to allow the user to edit the
cloud.yaml - where you will point the image tag to your local
custom image….

34. Changes to Cloud.yaml
keystone:
all:
+ image_full: kolla/centos-source-keystone:4.0.2
horizon:
all:
+ image_full: kolla/centos-source-horizon:4.0.2
cinder:
all:
+ image_full: kolla/centos-source-cinder-base:4.0.2

35. Once OpenStack is up...
● Lets confirm we actually orchestrated with local images
○ kubectl get pods --all-namespaces | grep horizon
○ kubectl describe pod horizon-blah -n kolla | grep Image
● And/or connect to your container and look for that proprietary code
○ docker run -tu root -ti horizon-blah /bin/bash
○ kubectl -n kolla exec -it horizon-blah -- ls

36. So you make some changes to your source
code..
● Repeat the steps to build the image
● Kill the container: kubectl delete pod horizon-blah -n kolla
● WHAT NEXT???

37. And commence OpenStacking...
● At this point you have a running Kolla-Kubernetes OpenStack running
with custom containers where needed.

38. Building and Orchestrating
DEVELOPMENT
Local Source Code
ORCHESTRATION and OPERATIONS
Neutron Nova
Cinder
GlanceHeat
Ironic
Keystone Horizon
KOLLA BUILD
Tools to build
containers
src/Keystone
src/Horizon
src/Cinder
Helm upgrade /
update config
Trash and restart
src/<Service>
Keystone Image
Horizon Image
Cinder Image
<Service> Image
Kube
Schedul
er
Kube
Controll
er
Kube
SDN
Kube
DNS
Kube
etcd
Kube
Proxy
Tiller Kube
API
KUBERNETES

39. Building and Orchestrating
DEVELOPMENT
Local Source Code
ORCHESTRATION and OPERATIONS
Kolla-Kubernetes
Neutron Nova
Cinder
GlanceHeat
Ironic
Keystone Horizon
KOLLA BUILD
Tools to build
images
src/Keystone
src/Horizon
src/Cinder
Helm upgrade /
update config
src/<Service>
Keystone Image
Horizon Image
Cinder Image
<Service> Image
Kube
Schedul
er
Kube
Controll
er
Kube
SDN
Kube
DNS
Kube
etcd
Kube
Proxy
Tiller Kube
API
Helm delete / Helm install