1.
#ibmedge© 2016 IBM Corporation
Continuous Integration with
Cloud Foundry Concourse and
Docker on OpenPOWER
Yulia Gaponenko yulia.gaponenko1@de.ibm.com
Indrajit Poddar ipoddar@us.ibm.com
IBM Systems
Sep 22, 2016

2.
#ibmedge
About Indrajit (a.k.a. I.P)
Expertise:
• Accelerated Cloud Data Services, Machine
Learning and Deep Learning
• Apache Spark, TensorFlow… with GPUs
• Distributed Computing (scale out and up)
• Cloud Foundry, Spectrum Conductor, Mesos,
Kubernetes, Docker, OpenStack, WebSphere
• Cloud computing on High Performance Systems
• OpenPOWER, IBM POWER
1
Indrajit Poddar
Senior Technical Staff Member,
Master Inventor, IBM Systems
ipoddar@us.ibm.com
Twitter: @ipoddar

3.
#ibmedge
About Yulia
2
Yulia Gaponenko
Software Engineer, IBM Systems
yulia.gaponenko1@de.ibm.com
github: barthy1
Expertise:
• Cloud Foundry committer (BOSH project)
• Distributed Computing (scale out and up)
• Cloud Foundry, Docker, OpenStack, WebSphere
• Open Source
• Linux, Eclipse
• Cloud computing on High Performance Systems
• OpenPOWER, IBM POWER

4.
#ibmedge
Please Note:
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice
and at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product direction and it
should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal
obligation to deliver any material, code or functionality. Information about potential future products may not be
incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products
remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending upon
many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the
I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve results similar to those stated here.
3

5.
#ibmedge
Agenda
• Continuous Integration (CI) : What is it?
• OpenPOWER Systems: What are these?
• CI for Open Source Software on OpenPOWER: Why is it important?
• Cloud Foundry: What is it?
• Concourse: What is it?
• Scenario: Continuously build a Concourse project from GitHub
4

6.
#ibmedge
Change is the only constant…
5
Collaborative Open Source Development has to embrace change
e.g. Linux has 2,000 new developers and
gets 10,000 patches for each version
http://arstechnica.com/information-technology/2015/02/linux-has-2000-new-
developers-and-gets-10000-patches-for-each-version/

7.
#ibmedge
What is the elephant like?
6
Ubuntu
CentOS
Fedora
Intel - X86_64
Debian
POWER – ppc64le
ARM – arm64
https://en.wikipedia.org/wiki/Blind_men_and_an_elephant
Open Source Development has to embrace variations
Microprocessor Architectures
Linux OS Distros

8.
#ibmedge
Continuous Integration for Open Source Development
• Merge developer copies into shared
mainline several times a day
https://en.wikipedia.org/wiki/Continuous_integration
• Best Practices
• Continuous Build
• Continuous Test
• Continuous Deployment
• Tools:
• Jenkins
• Travis
• Concourse
• Many more…
7

9.
#ibmedge
IBM POWER Systems Cloud Performance on Open
Source
• Runs popular Linux Distros
• Runs Docker containers at a high density
per system
https://g01acxwass069.ahe.pok.ibm.com/cms/developerworks/linux/perfcol/cloud.html
8

10.
#ibmedge
OpenPOWER: Open Hardware for High Performance
9
Systems designed for
big data analytics
and superior cloud economics
Upto:
12 cores per cpu
96 hardware threads per cpu
1 TB RAM
7.6Tb/s combined I/O Bandwidth
GPUs and FPGAs coming…
OpenPOWER
Traditional
Intel x86
http://www.softlayer.com/bare-metal-search?processorModel[]=9

11.
#ibmedge
OpenPOWER Worldwide Development Community
10
continents
20+ technologies revealed
innovations under way100+
members200+
24 6countries
OpenPOWER is an open development community,
using the POWER Architecture

12.
#ibmedge
IBM OpenPOWER Systems with Open Source Software
11
Open Source Software works great on POWER
GoLang
And many more …

13.
#ibmedge
Continuous Integration for Platform-as-a-Service (PaaS)
• Develop, run, manage applications
without worrying about the complexity of
the infrastructure
• Often runs on top of a Infrastructure-as-a-
Service e.g. OpenStack
• Examples:
• Apache Mesos, Kubernetes with Docker
e.g. docker run –t ppc64le/ubuntu
• Cloud Foundry
e.g. cf push helloworld.js
• CI and PaaS naturally go hand in hand
12

14.
#ibmedge
Continuous Integration for OpenPOWER
How to ensure that latest versions of open source
software packages always run on OpenPOWER?
• Source code changes from multiple developers do not cause
breakages
• Particularly when development is done on x86
• Changes are automatically tested on POWER architecture
• Binary packages for POWER are always available at the
most recent level
• in repository as binary blob
• as Docker image
• Automatic versioning
13
Docker
image
Source
code
Github
Building
next version
Testing on
Power
Repository

15.
#ibmedge
Concourse: Continuous Integration for Cloud Foundry
14
• Pipeline-based CI system written in Go.
• Website: concourse.ci

16.
#ibmedge
15
Cloud Foundry. What is it?
• Open source cloud computing Platform-as-a-Service (PaaS)
• Faster and easier to build, test, deploy, and scale applications
• Choice of clouds, developer frameworks, and application services.
Pivotal Cloud Foundry, HP Helion, Predix
GoLang
Commercial PaaS products based on open source Cloud Foundry:

17.
#ibmedge
Scenario: Continuously Build a Concourse project from GitHub
• https://github.com/barthy1/git-resource
• tracks commits in a branch of a Git repository
• Get:
• Github project (git-resource)
• Docker image with git (git-image)
• Put:
• New Docker image with git-resource to Docker registry (git-resource-image)
• Action:
• Build and test (git-resource)
16

18.
#ibmedge
Scenario: CI Tools
• Docker Registry: to distribute container images
• Open Stack: to create connected virtual machines on multiple servers
• BOSH: to deploy Cloud Foundry releases on Open Stack
• Github: to collaboratively develop Open Source Software
• Concourse: to continuously integrate open source development and test
– Concourse cli called fly
• Concourse CI descriptor, called a pipeline:
to design and execute a workflow for continuous integration
17

19.
#ibmedge
Scenario: CI Tools Interaction
18
Bare metals
VM VM
Concourse
Server
Docker Containers
VM VM
BOSH
Image Registry
Object Store
Concourse
fly client or
browser client
DJ1. Sets up CI infrastructure
DJ2. Adds CI pipeline
DJ3. Commit code
CS1. Monitor
CS2. Build
CS3. Test
4. Output
Developer Jane

20.
#ibmedge
1. Deploy your own or use a managed OpenStack
• Oregon State University Open Source Lab OpenStack
on OpenPOWER systems
• SoftLayer bare metals with Ubuntu OpenStack on OpenPOWER
19

21.
#ibmedge
2. Setup a Docker Private Registry
or use DockerHub
Docker Registry: Docker Image storage
DockerHub: Public Docker registry
• Docker private registry
• Install Docker
• Clone https://github.com/barthy1/distribution
• Build Docker image
• Configure TLS and basic auth
• Run Docker private registry
20

22.
#ibmedge
3. Install BOSH
BOSH: Tool to simplify deployment of other projects (Concourse, CF…)
• Deploy project using release packages
• Update deployment with new versions of the release packages
• Monitor state of the deployment
• Fix problems with VMs or connectivity
• Scale out with additional VMs
• Bosh installation:
• Binaries available from:
ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/
– bosh-init - BOSH installer
– BOSH release
– bosh cli – interact with BOSH server
• Sample deployment manifest:
https://www.cloudfoundry.org/bosh-on-openpower-systems/
– update only OpenStack connection details
and links to binaries
21

23.
#ibmedge
4. Install Concourse
Concourse: Project to continuously run tests and build binaries
• Concourse contains 2 releases:
• Concourse
• Garden-runc
• Concourse installation:
• Binaries available from:
ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/
• Sample deployment manifest
• Installation steps:
https://www.cloudfoundry.org/concourse-release-for-openpower
– upload all releases
– upload deployment manifest
– start deployment
22

24.
#ibmedge
5. Concourse pipelines
Pipeline: Set of steps to run the tests and build the result
• Various input and output data types
– Git - pull and push to git repositories
– Time - start on schedule or timestamp output
– Docker-image - fetch, build and push Docker images
– Semver – set or bump version numbers
– Amazon S3 - fetch from and upload to s3 buckets
– ….
– Custom resources
• Color of the box shows type and state
– Black – resources
– Green – successfully finished most recent build
– Red – test/build failed
• All the tests and the logs are available
• Group by tests for every commit
• History of the Github project updates
23

25.
#ibmedge
5. Examine an existing pipeline
• Pipeline sample:
• Input resources:
– Github project
– Docker image
• Test/build stage
– Clone/update Github project
– Pull/refresh Docker image
– Run the test
– Build new Docker image
– Push Docker image to Docker registry
• Output resources:
– New Docker image
• Logs are available for public/private access
• New test can be started via Web
• Access to the container/s with tests
• Information about current commit
24

26.
#ibmedge
LIVE DEMO!
25

27.
#ibmedge
6. Update the existing pipeline
• Existing pipeline to show live how it works
• Update the pipeline descriptor with new parameters
• Upload new pipeline to the server with Concourse cli
• Run new pipeline
• Show execution steps and logs
• Login to the running container with tests
• Run latest Docker image for POWER
26

28.
#ibmedge
Summary
• Continuous Integration is essential for open source projects
• CI benefits OpenPower
• Concourse tool is great for CI!
27

29.
#ibmedge
Resources
• Pointers to blogs on how to deploy BOSH, Cloud Foundry, Concourse on
OpenPOWER
http://blog.altoros.com/cloud-foundry-on-ibm-power.html
https://www.cloudfoundry.org/bosh-on-openpower-systems/
https://www.cloudfoundry.org/cf-release-for-openpower/
https://www.cloudfoundry.org/concourse-release-for-openpower
• How to setup Docker private registry:
https://gist.github.com/barthy1/246cd6892d5db2cec9b6f866de70c089
• Latest Docker binaries for POWER:
https://master.dockerproject.org/
• Latest Cloud Foundry binaries for POWER:
ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/
• Oregon State University Open Source Lab with OpenStack:
http://osuosl.org/services/powerdev
28

30.
#ibmedge
Notices and Disclaimers
29
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our
warranty terms apply.”
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or
their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

31.
#ibmedge
Notices and Disclaimers Con’t.
30
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

32.
© 2016 IBM Corporation #ibmedge
Thank You