Over the last 12 months at Auto Trader, we have been focusing our energy on moving our on premise workloads to Google Cloud Platform, and that includes our database architecture. Join me as we explore how we have migrated from on premise MongoDB clusters to a microservice aligned database architecture on MongoDB Atlas using Infrastructure as Code, and how we are integrating MongoDB into the Auto Trader Delivery Platform.

1. 1
Moving a Monolith to
MongoDB Atlas
Mohsin Patel
Principal Database Engineer, Auto Trader UK
@dbamohsin dbamohsin.wordpress.com

2. Over the last 12 months at Auto Trader, we have been
focussing our energy on moving our
on premise workloads to Google Cloud Platform, and
that includes our database architecture
Explore how we have migrated from monolithic on
premise MongoDB clusters to a micro service aligned
database architecture on MongoDB Atlas using
Infrastructure as Code, and how we are integrating
MongoDB into the Auto Trader Delivery Platform

3. A bit about me
{ 11 years }
{ 9 years }{ 14 years } { 11 years }
Infrastructure Squad – Multi-disciplined
Database
Engineers
System
Engineers
Network
Engineers
Developers Tech Lead

4. Our Story: 40 years in the making
4
1977
1995
UK national coverage of
Auto Trader magazines
2007
Implements digital
transition strategy
Digital revenue = Print revenue
2010
Websites for Retailers
Launches apps
2013
Closure of print magazines
2013
100% digital business
in UK and Ireland
Magazine Business
1996
Start of online operations -Launch of
autotrader.co.uk
Diversification
Transition to Digital
Fully Digital Business
1994 2007 2013

5. A Technology-focused business

6. The UK’s largest digital automotive marketplace…
6
£355.1m
Revenue
£5.5bn
Market Cap
831 Employees
FTSE 100
Company
HQ – Manchester
Offices in London
and Dublin
212 Developers
In
Multidisciplined
squads
monthly cross-
platform visits
50m 71%
of all time spent
on Auto classified
sites
is spent on
Auto Trader
94
searches
every second
9m
monthly
cross-
platform
unique users

7. Where we are…
2
physical
data
centers
269
apps deployed
in the public
cloud
3
public
clouds
455
Live
releases in
one day
15000
releases to
live in FY19
99.79%
release
success
rate
99.99%
availability of Auto
Trader search during
FY19 (Apr 18 – Apr 19)
441
apps in
total

8. Monolithic Architecture
[mono·lith·ic]
too large, too regular, or
without interesting differences,
and unwilling or unable to be changed:

9. Auto Trader database estate
0
10
20
30
40
50
60
70
80
90
%ofDatabases
2009
2019

10. On Premise MongoDB Clusters
• Architected in 2009
• Upgraded to 2.6.3 in 2014
• Upgraded to 2.6.8 in 2015 with the aim to
move to 3.0!
• Introduced Auth in 2015
DC1 DC2
1hr
Slave Delay
Priority
10
Priority
10
Priority
5
Priority
5
800GB 800GB 800GB 800GB 800GB
32GB RAM32GB RAM32GB RAM32GB RAM16GB RAM
• 3 Node cluster v2.6
• 4 Node cluster (Plus Arbiter) v3.2
• 3 Node cluster hosted in our private cloud v3.0
• Plus all the associated QA/DEV clusters
Other Clusters
26 VM’s 272 GB of RAM 8 TB’s of NetApp flash storage
30+ Application
Databases

11. Problem Areas
Sizing OpLogBlast Radius
Troubleshooting
Issues
Version 2.6
Aligning dev/test
resources
Maintenance Windows

12. Migrating to Atlas on GCP

13. Iterative Architecture
• A few different approaches before Atlas…
Containers in k8 using
Stateful Sets
Cloud Manager +
Chef Cookbooks
Ops Manager MongoDB K8
Operator
{ 2017 }
{ 2017 }
{ 2018 }
{ 2018 }

14. Iterative Architecture
• Back in November 2017..

15. Atlas at Auto Trader
• Atlas is a cloud agnostic Managed Database Service provided by MongoDB
• We use Atlas to provision database clusters within Google Cloud Platform
• Started proof of concept work in August 2018
Smaller and more manageable
clusters with closely knit application
databases
Upgraded architecture to a
minimum MongoDB version of 3.6
A highly scalable document store
infrastructure with increased
performance visibility
What does our platform look like?

16. Migration Methodology
Proof of
Concept
Latency Infra testing I/O Perf.App Complexity
Restoring
Data
Time boxed Data
Migration
Test Atlas
Features
Database
Analysis
Data
Dependencies
Data Purging
Database
Ownership
Logical
Grouping Sizing Estimate
Infrastructure
Analysis
DB Version
Backup
Policy
Networking
Cluster
Sizing Regions
Budget
Forecast
Application
Analysis
I/O Profile Busy PeriodsConnectivity
Connection
Pooling
Downtime
Driver
versions
Impact of App
Scaling
On a per app basis, as we migrate…

17. Migration Methodology: Migration Flow
Push App Changes PIT/QA
Purge Unneeded
Data
Pick an App
Candidate
Database Analysis
Infrastructure
Analysis
Monitor service
health metrics
Application Analysis
Update Driver >=
3.6
Convert
Connections to URI
Preprod Data
Migration
Export/Import
Live migrate
App to DB Auth
Prevent on Prem
Access
Arrange Prod
migration window
Prod Migration
Migrate!

18. Migration Methodology: Decom
Monitor mongo
logs
Remove User
Access
Database Analysis
Infrastructure
Analysis
Application Analysis
Monitor Kibana
Logs for app
Wait ~7 daysWait ~7 Days
Drop database
from preprod Migrate!
Drop database
from Prod
Reclaim Space – if
needed
Decom Cluster
nodes if last DB
Decom

19. Benefits
19
Platform
App Development
Availability
Monitoring
IAC using Terraform Scalability
SSL as default MongoDB+SRV Atlas Monitoring Alerting Built In
Regional Replication
Automated
Backups/ Queryable

20. MongoDB Atlas Migration Progress
databases in prod
12 Production Clusters
migrations from on premise clusters
Virtual Machines GB of RAM Terabytes of flash
90% Migrated in 6-9 Months!
32 28
Decom value:
26 272 8

21. 6 Migration Lessons Learnt

22. Driver Versions
Preparing applications for migration
1

23. Connection Pooling
Atlas Clusters have connection limits
2

24. Infrastructure As Code Quirks
Conflicts between enabling good Atlas features such as auto disk
scaling versus defining in code
3

25. Database Users
Users are created at project level, not per cluster
4

26. IP Whitelisting
Application IPs can be ephemeral in a non VPC cloud
architecture model
5

27. Data Purging Policies
Backups are a variable cost – control the data
6

28. Unified Delivery Platform

29. Providing a capability driven platform
that all of our services can be built on

30. MongoDB Integration into the platform
• All of our platform capabilities are exposed via simple configuration values
• A couple of values drive our mongodb integration
• Every time an application is committed, configuration is validated and deployed to our Delivery Platform
(Kubernetes)
• These values are then realized into true infrastructure
• If mongodb enabled is true then give this app mongo metrics
Or
• If mongodb is enabled and a cluster hasn’t been provided, then fail this deployment

31. 31

32. 32

33. MongoDB Integration into the platform

34. Driving Observability
MongoDB Exporter

35. Driving Metric Visibility
• Give developers access to mongo cluster metrics as part of their application deployment

36. MongoDB Grafana Dashboard

37. Platform alert forwarder
• Skippr is an inhouse alert forwarder
• Used extensively for our Delivery Platform
• Now working on integrating MongoDB via an
Atlas webhook alert service
• See ownership of alerts in squad channels
• Quickly see if the database behind an
application is having issues

38. Infrastructure as Code
• Create repeatable code
• Ensure all clusters follow same
standards
• DB infrastructure sits in same
repository as rest of delivery
platform
• Helps prevent manual changes via
the GUI

39. Key Takeaways
Quick to get up and running on Atlas
Platform integrations make it easier for our
developers to consume MongoDB
Spend more time on proactive business value

40. Looking ahead

41. Enhancing Observability of the
platform

42. 42
DEMO

43. 43

44. Unified Billing
Billed via Google for our Atlas usage

45. Switching to the official terraform
provider for MongoDB Atlas

46. Auto Trader MongoDB Cli
Command Line Interface to let us talk to the MongoDB API

48. We are Hiring!
@autotrader_life
autotrader_life
careers.autotrader.co.uk
auto-trader-uk
autotraderuk

49. #MDBlocal
Migrating a Monolith to
MongoDB Atlas - Auto
Trader’s Journey [Auto
Trader]
Mohsin Patel
https://www.surveymonkey.com/r/K5LPNK2