This document discusses advanced replication in MongoDB replica sets. It begins by explaining the roles and configuration of replica set members, including primaries, secondaries, and arbiters. It then covers the implementation details of replication using an oplog and heartbeats between nodes. The rest of the document discusses best practices for maintenance, data center awareness using tagging, write concerns, and read preferences to control data distribution and failover.

1. #MongoDBDays - @m4rcsch
Advanced Replication
Marc Schwering
Solutions Architect, 10gen

2. Notes to the presenter
Themes for this presentation:
• Prepare your console use properly

3. Roles & Configuration

4. Node 1 Node 2
Secondary Arbiter
Heartbeat
Re
pli
cat
on i
Node 3
Primary
Replica Set Roles

5. Configuration Options
> conf = {
_id : "mySet",
members : [
{_id : 0, host : "A"},
{_id : 1, host : "B"},
{_id : 2, host : "C", "arbiter" : true}
]
}
> rs.initiate(conf)

6. Simple Setup Demo

7. Behind the Curtain

8. Implementation details
• Heartbeat every 2 seconds
– Times out in 10 seconds
• Local DB (not replicated)
– system.replset
– oplog.rs
• Capped collection
• Idempotent version of operation stored

9. Op(erations) Log

10. Op(erations) Log is idempotent
> db.replsettest.insert({_id:1,value:1})
{ "ts" : Timestamp(1350539727000, 1), "h" :
NumberLong("6375186941486301201"), "op" : "i", "ns" :
"test.replsettest", "o" : { "_id" : 1, "value" : 1 } }
> db.replsettest.update({_id:1},{$inc:{value:10}})
{ "ts" : Timestamp(1350539786000, 1), "h" :
NumberLong("5484673652472424968"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 },
"o" : { "$set" : { "value" : 11 } } }

11. oplog and multi-updates

12. Single operation can have many
entries
> db.replsettest.update({},{$set:{name : ”foo”}, false,
true})
{ "ts" : Timestamp(1350540395000, 1), "h" :
NumberLong("-4727576249368135876"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 2 }, "o" :
{ "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 2), "h" :
NumberLong("-7292949613259260138"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 3 }, "o" :
{ "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 3), "h" :
NumberLong("-1888768148831990635"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 }, "o" :
{ "$set" : { "name" : "foo" } } }

13. Operations

14. Maintenance and Upgrade
• No downtime
• Rolling upgrade/maintenance
– Start with Secondary
– Primary last
– Commands:
• rs.stepDown(<secs>)
• db.version()
• db.serverBuildInfo()

15. Upgrade Demo

16. Replica Set – 1 Data Center
• Single datacenter
Datacenter
• Single switch & power
Member 1 • Points of failure:
– Power
Member 2 – Network
– Data center
Datacenter 2
Member 3 – Two node failure
• Automatic recovery of
single node crash

17. Replica Set – 2 Data Centers
• Multi data center
Datacenter 1
• DR node for safety
Member 1
• Can’t do multi data
Member 2 center durable write
safely since only 1 node
in distant DC
Datacenter 2
Member 3

18. Replica Set – 2 Data Centers
• Analytics
• Disaster Recovery
• Batch Jobs
• Options
– low or zero priority
– hidden
– slaveDelay

19. Replica Set – 3 Data Centers
Datacenter 1 • Three data centers
Member 1
Member 2
• Can survive full data
center loss
Datacenter 2
Member 3 • Can do w= { dc : 2 } to
Member 4 guarantee write in 2
data centers (with tags)
Datacenter 3
Member 5

20. Replica Set – 3+ Data Centers
Secondary
Primary
Secondary
Secondary
Secondary delayed
Secondary
Secondary

21. Commands
• Managing
– rs.conf()
– rs.initiate(<conf>) & rs.reconfig(<conf>)
– rs.add(host:<port>) & rs.addArb(host:<port>)
– rs.status()
– rs.stepDown(<secs>)
• Minority reconfig
– rs.reconfig( cfg, { force : true} )

22. Options
• Priorities
• Hidden
• Slave Delay
• Disable indexes (on secondaries)
• Default write concerns

23. Developing with Replica Sets

24. Client Application
Driver
Write
Read
Primary
Secondary Secondary
Strong Consistency

25. Client Application
Driver
Write
ad
Re
Re
ad
Primary
Secondary Secondary
Delayed Consistency

26. Write Concern
• Network acknowledgement
• Wait for error
• Wait for journal sync
• Wait for replication
– number
– majority
– Tags

27. Write Concern Demo

28. Datacenter awareness (Tagging)
• Control where data is written to, and read from
• Each member can have one or more tags
– tags: {dc: "ny"}
– tags: {dc: "ny",
subnet: "192.168",
rack: "row3rk7"}
• Replica set defines rules for write concerns
• Rules can change without changing app code

29. Tagging Example
{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})
> db.getLastErrorObj({"someDCs"})

30. Driver
getLastError
W:allDCs
write
Primary (SF)
replicate
apply in
memory
Secondary (NY)
replicate
Secondary (Cloud)
Wait for Replication

31. Write Concern with timeout
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.getLastErrorObj({"allDCs"},100);
> db.getLastErrorObj({”someDCs"},500);
> db.getLastErrorObj(1,500);

32. Read Preference Modes
• 5 modes
– primary (only) - Default
– primaryPreferred
– secondary
– secondaryPreferred
– Nearest
When more than one node is possible, closest node is used for
reads (all modes but primary)

33. Tagged Read Preference
• Custom read preferences
• Control where you read from by (node) tags
– E.g. { "disk": "ssd", "use": "reporting" }
• Use in conjunction with standard read preferences
– Except primary

34. Tags
{"dc.va": "rack1", disk:"ssd", ssd: "installed" }
{"dc.va": "rack2", disk:"raid"}
{"dc.gto": "rack1", disk:"ssd", ssd: "installed" }
{"dc.gto": "rack2", disk:"raid”}
> conf.settings = { getLastErrorModes: { MultipleDC :
{ "dc.va": 1, "dc.gto": 1}}
> conf.settings = {
"getLastErrorModes" : {
"ssd" : {
"ssd" : 1
},...

35. Tagged Read Preference
{ disk: "ssd" }
JAVA:
ReadPreference tagged_pref =
ReadPreference.secondaryPreferred(
new BasicDBObject("disk", "ssd")
);
DBObject result =
coll.findOne(query, null, tagged_pref);

36. Tagged Read Preference
• Grouping / Failover
{dc : "LON", loc : "EU"}
{dc : "FRA", loc : "EU"}
{dc : "NY", loc : "US”}
DBObject t1 = new BasicDBObject("dc", "LON");
DBObject t2 = new BasicDBObject("loc", "EU");
ReadPreference pref =
ReadPreference.primaryPreferred(t1, t2);

37. Tagging Demo

38. Conclusion

39. Best practices and tips
• Odd number of set members
• Read from the primary except for
– Geographically distribution
– Analytics (separate workload)
• Use logical names not IP Addresses in configs
• Set WriteConcern appropriately for what you are
doing
• Monitor secondaries for lag (Alerts in MMS)

40. #MongoDBDays - @m4rcsch
Thank You
Marc Schwering
Solutions Architect, 10gen