Dc migration

1. Merging two live data-centers into one
Anush Arvind
Senior DevOps + SRE + Infrastructure Engineer, Freshworks

2. The problem
Frankfurt
GDPR
Cost
Maintenance
Source: Google images

3. The idea
Merge & Migrate
IrelandFrankfurt

4. The idea
Merge & Migrate
Source: Google images

5. The challenges
High volume of tickets in both DC’s
Migration of all components of our infrastructure
Data collision
Seamless migration for customers
Multiple services for similar data sources

6. The Solution
Migration
notice
VPC
Peering
Storage
in S3
Database
ELB
Redis
Elasticache
server
Scalability of
destination
DC

7. Migration notice
Migration notice to
Ireland Customers
Change DNS & IP
listing to Frankfurt DC
Try and ensure
execution on time

8. VPC Peering
Ireland FrankfurtPrivate IP

9. Storage in S3
Storage
In
Ireland
Storage
In
FrankfurtMigrate
Existing
Storage
Reroute
new
storage

10. Database
Horizontally
partitioned and
no ID clash
Ireland Frankfurt

11. Database
When there is ID clash
Source: Google images

12. ELB
Ireland Frankfurt
Migrate
Source DC
certificates

13. Redis
Replication was not possible
Source: Google images

14. The solution

15. The solution

16. Scale up
Frankfurt
Frankfurt

17. The Migration
Maintenance
Banner in
Ireland DC
Verify if incoming
traffic and writes
have stopped in
Ireland DC
Stop replication of
data stores and
Make Frankfurt as
Master
Send some
traffic to
Frankfurt and
test
Point all traffic
to Frankfurt DC

18. Surprise!
Some customers had still not
updated their DNS
Vanity URL would go to Ireland,
take an extra hop and come to
Frankfurt
Source: Google images Source: Google images

19. Post migration
Ireland Bring down source DC
Document
Source: Google images

20. Learnings
Run multiple poller scripts to
replicate live
Frankfurt should accept its own
and Ireland’s API keys
Warm up databases before
pointing it to production
Source: Google images

21. Conclusion
On a broader level, we had some interesting learnings for any project.
On the whole, it was a tedious yet interesting project to migrate our DC and am
happy to have shared the story with you all.
Never be surprised
by a new challenge
Always have plan B Be ready to rollback
Source: Google images

22. anushshiva
anush.arvind@freshworks.com
https://in.linkedin.com/in/anush-arvind-0b5844a5

23. Questions?