My talk at OSCON 2012.

1. Running MongoDB for
High Availability
Greg Brockman
Stripe
twitter.com/thegdb
gdb@stripe.com

2. Overview
• Intro
• Tips & Tricks
• Best Practices
• Questions

3. Traditional DB
Availability

4. When the master fails...

5. Replica Set Crash
Course
PRIMARY SECONDARY
WRITE
READ SECONDARY

6. Replica Set Crash
Course
DEAD SECONDARY
WRITE !!
READ SECONDARY

7. Replica Set Crash
Course
DEAD SECONDARY
WRITE
READ PRIMARY

8. If at first you don’t
succeed...

9. How it works in
practice

10. Without retries
zz

11. With retries

12. All the (hard) problems
are solved

13. Tips & Tricks

14. Force initial syncs from
secondaries
newnode> iptables -I OUTPUT -j DROP -d master
# new in 2.2
newnode> rs.syncFrom(...)

15. Rebuild whole clusters
at a time
Old
New

16. Rebuild whole clusters
at a time
Old
New

17. Rebuild whole clusters
at a time
> rs.stepDown()
> rs.freeze(60)

18. Rebuild whole clusters
at a time

19. Rebuild whole clusters
at a time

20. Rebuild whole clusters
at a time

21. Rebuild whole clusters
at a time
> rs.stepDown()
> rs.freeze(60) > rs.freeze(60)
> rs.freeze(60) > rs.freeze(60)

22. Rebuild whole clusters
at a time

23. Rebuild whole clusters
at a time

24. Consider using IPs vs
hostnames

25. Do zero-downtime
migrations
Use object Avoid multi-
versioning updates!

26. Create indexes online
• Build in background
• Build one node at a time
$ mongod --port 8000

27. Best Practices

28. Always rehearse your
exact procedure

29. Be skeptical

30. Read the issues

31. Understand the
guarantees

32. Keep a record of how
you’ve been burned

33. Greg Brockman
Stripe
twitter.com/thegdb
gdb@stripe.com