Conductor has built an automated CI and CD process which has allowed us to test and deploy high-quality code quickly and reliably. During this presentation, we demonstrated how we leveraged Docker, AWS, TeamCity and other modern technologies to improve and streamline our development process. We also discussed the challenges we face as we shift away from a monolithic build to a microservice architecture.
5. 5
“The practice of integrating and testing source code continuously
in an automated fashion”
WHAT IS CONTINUOUS INTEGRATION?
6. 6
Minimize the duration and effort required by each integration
episode
Be able to deliver a product version suitable for release at any
moment
OBJECTIVES
7. 7
Maintain a single source repository
Automate the build
Make Your build self-testing
Keep the build Fast
Test in a clone of the production environment
Automate deployment
PRACTICES OF CONTINUOUS INTEGRATION
10. 10
· Choose and setup a CI tool
· Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS
11. 11
Choose and setup a CI tool
· Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS
14. 14
Choose and setup a CI tool
Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS
15. 15
Easy to spin up and maintain
Immutable and disposable Infrastructure
Production-like
Cost effective
Environment agnostic
TEST ENVIRONMENT REQUIREMENTS
19. 19
Choose and setup a CI tool
Build and configure Testing Environment
Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS
20. 20
PULL REQUEST WORKFLOW
Open PR
Unit and
Integration
Tests
Jasmine and
Karma Tests
Linting
Launch
Selenium Grid
Launch PBE Feature Tests
Merge PR
22. 22
Choose and setup a CI tool
Build and configure Testing Environment
Build a Continuous Integration pipeline
Collect statistics and metrics
MILESTONES / GOALS
23. 23
“require branches to be up to date before merging” is disabled in Github
To mitigate conflicts we run a “heartbeat” workflow against master every 4 hours
Posts build status into Slack dev channels
PRs are paused in the case of failure
PR HEARTBEAT
26. 26
Choose and setup a CI tool
Build and configure Testing Environment
Build a Continuous Integration pipeline
Collect statistics and metrics
MILESTONES / GOALS
28. 28
Inconsistent behavior (flaky tests, code, infrastructure, etc.)
Differences in production and PBE environments
AWS is having a bad day
COMMON BUILD PROBLEMS
29. 29
FLAKY TESTS AND HOW WE MITIGATE THEM
Validate
artifacts
Launch
Selenium
Grid
Launch PBE
Rerun
Failures
Terminate
Resources
Green Check
PR
30. 30
Kubernetes is an open-source
system for automating deployment,
scaling, and management of
containerized applications.
It groups containers that make up
an application into logical units for
easy management and discovery.
TEST ENVIRONMENTS ON KUBERNETES
Kubernetes Cluster
Selenium Grid
Hub
Node
Node
Node
Searchlight
ZooKeeper
WireMock MySQL
Thrift
PBE Selenium Grid
Hub
Node
Node
Node
Searchlight
ZooKeeper
WireMock MySQL
Thrift
PBE
31. 31
TC CLOUD AGENTS
Configured TeamCity with AWS to start and stop images with TeamCity agents on-
demand based on the queued builds.
33. 33
Tests similar to those run for PRs are used
Runs on a production-like beta environment
Test results are reviewed
Jenkins job deploys in rolling fashion with zero downtime
DEPLOY TESTING
36. 36
Integrate performance tests into the PR process
Run security scanning as part of PRs
Remove reliance on release testing
Microservices!
WHAT’S NEXT?
37. 37
Large Monolithic applications are complicated!
They create unnecessary dependencies
They are hard to scale vertically and horizontally
Overall velocity and speed is hindered
WHY MICROSERVICES?
38. 38
Define organizational standards for designing, developing, testing and
deploying microservices
Use out of the box solutions as much as possible
Build new features as microservices alongside the monolith
Migrate existing features to microservices when necessary
GREAT, NOW WHAT?
39. 39
MICROSERVICE CI/CD
Jenkins
Dev Kubernetes Cluster
(1) a PR is opened against
a branch in GitHub
(2) Jenkins runs tests against that branch, if
successful changes are merged
(3) Jenkins runs tests against the head of features, if successful
artifacts are published to Nexus and the Docker registry
(4) Engineers can deploy their service manually to the development
Kubernetes cluster for manual validation during development
Prod Kubernetes Cluster
(5) Deployments to the production Kubernetes cluster are
triggered manually
E2E TESTS PERF TESTS
SCALE
TESTS
(6) E2E, performance and scale tests are run in a "blue"
environment and if successful the service is switched to "green"
Nexus
Docker
Registry
CD Tool
LINTING UNIT TESTS
CONTRACT
TESTS
CODE
COVERAGE
PUBLISH
ARTIFACTS
GitHub
Jenkins
LINTING UNIT TESTS
CONTRACT
TESTS
CODE
COVERAGE
OPEN PR
40. 40
· Daily monolith deploys
· Frequent microservice deploys
· Deliver bug fixes and new features faster
· Fewer bugs in production
· Continued adoption of microservices
WHERE WE ARE TODAY