This document discusses the process of transitioning a legacy monolithic application called San Diego to a microservices architecture with continuous deployment. The key steps taken included:
1) Identifying domain objects and services and using a strangler pattern to gradually migrate functionality from the monolith to new services.
2) Implementing continuous integration, delivery, and deployment practices like Docker containers, automated testing, and deploying on every code change.
3) Defining processes around pair programming, code quality, feature flags and dashboards to monitor the new system.
4) Using automation to deploy new versions, replacing containers on load balancers to achieve near real-time deploys with minimal downtime.
3. THIS TALK
▸ Background
▸ The approach
▸ Process / standards
▸ Build pipelines
▸ Results & lessons learned
4.
5. THE SYSTEM - SAN DIEGO
▸ ... or the Big Ball Of Mud
▸ Large legacy monolith
▸ Generates significant income
▸ Slow & complex
▸ Technical debt
6. SAN DIEGO
FRONTEND
MYSQL
DB
SAN DIEGO
BACKEND
LOAD BALANCERS / VARNISH
ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK
SAN DIEGO
FRONTEND
SAN DIEGO
FRONTEND
SAN DIEGO
FRONTEND
SAN DIEGO
BACKEND
SAN DIEGO
BACKEND
SAN DIEGO
BACKEND
MEMCACHE FTP
EXT.
SERVICES
SOLR
7. THE SYSTEM - SAN DIEGO
▸ Infrequent, manual releases
▸ Fragile tests
▸ Low velocity
▸ Frequent outages / bugs / issues
▸ Frustrated team
▸ Low confidence modifying existing code
15. STARTING OFF
▸ Scrum, 1 week sprints
▸ TDD / BDD
▸ Definition of Done
▸ Team mindset / experience
▸ Focus on value
▸ Replace old features with new (legacy becomes obsolete)
34. DEFENSE IN DEPTH
UNIT TESTS
INTEGRATION
TESTS
ACCEPTANCE
UI TESTS
public function testJobCannotBeFound() {
$jobRepository = $this->prophesize(JobRepository::class);
$jobRepository->getById(EXPECTED_JOB_ID)
->shouldBeCalled()
->willReturn(false);
$jobService = new JobService($jobRepository->reveal());
$this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
}
35. DEFENSE IN DEPTH
UNIT TESTS
INTEGRATION
TESTS
ACCEPTANCE
TESTS
UI TESTS
public function testFindJob() {
$expectedJob = $this->loadFixture('active_job.yml');
$actualJob = $this->repository->getById($expectedJob->getId());
self::assertInstanceOf(Job::class, $actualJob);
self::assertEquals($expectedJob->getId(), $actualJob->getId());
}
36. DEFENSE IN DEPTH
UNIT TESTS
INTEGRATION
TESTS
ACCEPTANCE
TESTS
UI TESTSScenario: Link to related job
Given a job exists
And there are related jobs available
When that job is viewed
Then a list of related jobs is shown
And each related job links to the detail page of the related job
40. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
41. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker pull
42. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker run
43. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
wait_for: port=8080 delay=5 timeout=15
44. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
uri:
url: http://localhost:8080/_health
status_code: 200
timeout: 30
45. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2
dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
46. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2
dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
47. DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker stop
docker rm
48. PIPELINE AS CODE
node {
stage 'Run tests'
sh "phpunit"
sh "behat"
stage 'Build docker image'
sh "phing build"
sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
sh "docker push jobservice:${env.BUILD_NUMBER}"
stage 'Deploy acceptance'
sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i acc deploy.yml"
stage 'Deploy production'
sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i prod deploy.yml"
}
52. RESULTS
▸ Total build time per service < 10 minutes
▸ Significantly improved page load times
▸ Improved audience stats (time on page, pages per session,
session duration, traffic, seo ranking, etc)
▸ Increased confidence and velocity
▸ Experimented with new tech/stacks (angular, jvm, event
sourcing)
▸ More fun
53. LESSONS LEARNED
▸ Team acceptance
▸ Change is hard
▸ Overhead of weekly sprint; requires discipline
▸ Docker orchestration
▸ Issues with traffic between Amazon <-> on-premise
datacenter
▸ Javascript testing
54. LESSONS LEARNED
▸ Experience with new tech
▸ Stability of build pipelines
▸ Management/leadership buy-in
▸ Business alignment
▸ Not enough focus on replacing legacy application