Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
From java monolith to kubernetes microservices - an open source journey with jenkins x
1. Java Monolith to Kubernetes:
An Open Source Journey with Jenkins-X
Ryan Dawson and Mauricio Salatino
2. Outline
● Started with a monolith, single job
● Microservices = many jobs, many artifacts
● Kubernetes = various artifact types
● Jenkins-X loves k8s apps
● And it loves CD and gitops
● We also build libraries with it
● And it works pretty well!
We’re from activiti, we do open source business automation and business process management. We’re going to explain to you how we went from building our project as a monolith to microservices and how we found Jenkins-X and what we’re doing with it. We’ll talk about all the cool things Jenkins-X can do for you out of the box and how we’ve customised it.
Two years ago we on the Activiti project had a pretty standard setup for the time.
One repository, one build job.
One type of artifact - maven.
Build process driven by features in CI (bamboo)
Releases manually initiated
Microservices -> lots of different types of artifacts. Not just maven anymore.
Not just maven & nexus -> decisions
docker images: where to host? Where to run?
Where should the nexus live? Should the docker registry come from the cloud provider? Use helm? Where to host helm charts?
Jenkins-X answers many of these questions.
So Out of the box it is highly opinionated.
It adds all the glue to run CI/CD well in Kubernetes and follow best practices
Has a `jx create cluster` wizard - https://youtu.be/xN9eydj4SHQ?t=17 - concluding at https://youtu.be/xN9eydj4SHQ?t=146
We can see nexus and chartmuseum and so on https://youtu.be/xN9eydj4SHQ?t=213
JX is using Jenkins but isn’t just Jenkins.
It’s Kubernetes-native
Wizards for apps as well as clients
Jx import from cloned repo
Adds e.g. dockerfile based on draft pack
Adds jenkinsfile and creates pipeline
As well as importing existing apps, jx cli can create a new one from a template
And give us a pipeline for it - so let’s see that
https://youtu.be/xN9eydj4SHQ?t=250
So when I do `jx import`:
Draft pack is applied
Webhooks created on git repo
Jenkins pipeline created based on Jenkinsfile (added by draft pack if not present)
Main branch of pipeline runs (master by default)
Version tag created and artifacts built.
Maven artifact goes to nexus in the cluster
Docker image built and goes to cloud provider registry
Chart goes to chartmuseum in cluster
PR goes to environment
PR merged to environment and env chart deployed
JX creates namespaces out of the box for isolated environments.
By default you get a staging environment and your apps get deployed there - https://youtu.be/xN9eydj4SHQ?t=186
Env represented by a git repository containing a helm chart.
Big helm chart represents all the microservices = ‘umbrella chart’ or ‘environment chart’.
JX works in GitOps style - it uses git to represent environments.
So if you were to lose an environment, you could get it back - everything is tracked
JX puts as much in git as possible e.g. Jenkinsfiles
So if your CI system dies, you don’t lose your pipeline definitions
And a pipeline should run when git changes through webhook
CD approach in JX makes promotion as automated as possible, as much as possible
So every commit to master is tagged, versioned and built
And automatically promoted through PRs
Emphasis with CD is on getting to prod, not sheltering code away
Fixed promotion-ready versions, not temporary snapshots
Promotion through automation, not code freezes for manual tests
Only the production gate is manual by default (so continuous delivery, not continuous deployment)
To make sure the flow is crystal clear:
Create a PR to a microservice (could here check it through preview env)
Commit the change by merging it
Pipeline runs based on webhook - it creates a fixed version tag and builds a docker image and helm chart version.
Pipeline creates PR into the staging environment.
If mergeable then the staging environment gets updated.
There’s a similar promotion workflow from staging to production.
https://youtu.be/xN9eydj4SHQ?t=454
Actually the previous slide skipped over preview environments but they are optional
They are namespaces dedicated to an instance of a microservice for a PR
By default you just get that version of the microservice there - but you can add more or link to services in the staging environment
Having JX propose answers for lots of tooling decisions helped us a lot
It’s cool that all the whole workflow is kubernetes-native, from the use of gitops to build pods
It showed us how we could do CD for k8s
We wanted to be sure Activiti Cloud projects would able to do this
For us to do it too we realised we had to depart from some defaults
We are a vendor building open source libraries and components - so we needed hosting to be more public
So we had to override some things not in the wizards
Specifically in secrets in the JX k8s configuration.
We also wanted to run various types of end to end tests in different namespaces but we’ll come back to that bit later.
JX defaults are aimed at a building typical web apps.
Our project structure more complex than typical web app.
We’re a vendor producing libraries that people can build on top of. And we’re also building projects ourselves using those libraries.
Our pipelines to need chains of deps.
One pipeline needs to update another pipeline or several.
So not just an environment requirements file being updated by pull requests resulting from builds of applications.
Need PRs for maven artifacts etc.
PR targets could be other projects.
JX under the hood uses updatebot.
A java project - a jar file that has a jenkins plugin and you can use it in your pipelines.
Specify list of target/downtream repos in yaml file
Call update with version and type of artifact
It scans the target repos and sends the update there, creating PRs.
https://drive.google.com/file/d/1yM2DBP-1WeMpAkAovOp2QqJS_wjHbSh0/view?usp=sharing
Some projects consumed by several projects e.g. ‘activiti-cloud-dependencies’.
So its Jenkinsfile tells updatebot to push the version to the configured target repos
Updatebot goes through all target repos, clones each, determines where to put the version and creates a pull request.
This happens right after we’ve pushed the maven artifact - it is the same process for other types of artifact.
PR gets evaluated on the other end - the target project has its own configuration to run tests against any pull requests.
Here some of the downstream repos are controlled by different JX clusters.
Updatebot means lots of PRs.
We don’t want to be merging those manually.
And CD means reduce manual gates.
So we needed good tests in each of the repos.
mergify: PRs are automatically merged when tests and checks pass.
We build different sets of examples from our base libraries.
Need to test interactions of components per example with E2E tests
We represent each example with a helm cart, each with a dedicated pipelines.
Branches for those pipelines dynamically create namespace per PR.
E2E tests run against namespace and it is destroyed at the end.
Kinda like PRs into a staging environment but envs created and destroyed dynamically. Like preview envs but without having to do service linking to talk to other services.
Lots of pipelines
Multiple instances of apps with each app instance containing a bunch of different microservices and databases.
So we’ve got some fairly powerful clusters running this.
Jenkins-X uses an elastic pool of build pods to run the build jobs so it can scale up and down… but still needs a fair bit of power.
We chose multiple clusters for isolation of the different parts of our project for clarity and independent upgrading - we could’ve run it all on one cluster from a resourcing point of view. JX can also be multi-tenanted.
With all these PRs is that occasionally you get little github blips.
It’s so rare that you don’t normally notice it but we make loads of requests.
So we’ve had to put retries in and make sure we’re handling the odd failure.
General distributed systems considerations but here applied to the CI system - because it’s distributed.
A lot of activity going through these pipelines.
A couple of times we have hit github’s rate-limiting.
Currently we have a user dedicated to updatebot and we rarely hit the limit but if we grow much more then we’ll have to start using separate users for different concerns.
Some configuration appears in multiple places in our setup...especially helm charts.
At the library layer very little repetition.
Every app has a helm chart but some of the apps have pretty much the same chart. There are ways to deal with this but it requires thought and some tradeoffs.
We’re just using ‘jenkins-x classic’.
There’s another version of jenkins-x you can choose if you want that is built on an even more powerful model, with knative to do run jobs on kubernetes in a way that scales much like serverless.
https://medium.com/@jdrawlings/serverless-jenkins-with-jenkins-x-9134cbfe6870
We’re not quite ready to migrate to that yet but it’s certainly exciting and we’re keeping a close eye on it.
We’re also keeping an eye on options for prod rollout strategies, esp istio - https://blog.csanchez.org/2019/01/24/progressive-delivery-with-jenkins-x/
