The document summarizes Continuent Tungsten, an open source tool for clustering and replication of MySQL databases. It discusses Tungsten Replicator's capabilities including master-slave, multi-master, and advanced topologies. It provides examples of installing different replication topologies using Tungsten's command line tools and cookbooks for common deployment scenarios. Conflict prevention is discussed as a method for avoiding data inconsistencies in multi-master configurations.

1. Tungsten University:
Set Up and Manage
Advanced Replication Topologies
Robert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013

2. Introducing Continuent
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
©Continuent 2013 2

3. A Review of Tungsten Replicator
©Continuent 2013 3

4. Tungsten Replicator Overview
Master
Replicator THL
Download
transactions
via network (Transactions + Metadata)
DBMS
Logs
Slave Replicator THL
Apply using JDBC (Transactions + Metadata)
©Continuent 2013 4

5. Master Replication Service
Pipeline
Stage Stage
Extract Filter Apply Extract Filter Apply
Slave
Binlog
tcp/ip Replicators
In-Memory Transaction
MySQL History Log
Queue
Master
©Continuent 2013 5

6. Slave Replication Service
Pipeline
Stage Stage Stage
Extract Filter Apply Extract Filter Apply Extract Filter Apply
tcp/ip
Master
Replicator
Transaction In-Memory Slave
History Log Queue DBMS
©Continuent 2013 6

7. Overview of Installation Process
1. Set up hosts
2. Prepare MySQL replicas
3. Download software
4. Install using tungsten-installer & con"gure-
service
Amazon Setup:
https://docs.continuent.com/wiki/display/TEDOC/
Preparing+EC2+Servers
©Continuent 2013 7

8. How tungsten-installer Works for
Basic Master/Slave Deployment
Staging copy db1
of files
db2
check prereqs
copy code db3
configure
©Continuent 2013 8

9. From Master/Slave Replication ...
db2
Replicator
db1
Replicator Service alpha
Service alpha
db3
Replicator
Service alpha
tungsten-
installer
Install master and slaves on the whole cluster
©Continuent 2013 9

10. To Multi-Master
db2
db1 Replicator Replicator
Service alpha Service alpha
Service bravo Service bravo
tungsten- tungsten- con!gure- con!gure-
installer installer service service
Install master on install master install slave install slave
db1 on db2 service on db1 service on db2
©Continuent 2013 10

11. tungsten-installer master 1
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
./tools/tungsten-installer
--master-slave
--master-host=$MASTER1
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=alpha
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER1
--start
creating service 'alpha'
Notice: --cluster-hosts has only one host
©Continuent 2013 11

12. tungsten-installer master 2
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
./tools/tungsten-installer
--master-slave
--master-host=$MASTER2
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=bravo
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER2
--start
creating service 'bravo'
Notice: --cluster-hosts has only one host
©Continuent 2013 12

13. Con"gure Service master 1
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER1
--datasource=$MASTER1
--local-service-name=alpha
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER2
--master-thl-port=2112
--svc-start bravo
Notice: bravo is the master service in host 2
©Continuent 2013 13

14. Con"gure Service master 2
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER2
--datasource=$MASTER2
--local-service-name=bravo
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER1
--master-thl-port=2112
--svc-start alpha
Notice: alpha is the master service in host 1
©Continuent 2013 14

15. Advanced Installation
using Cookbooks
©Continuent 2013 15

16. From Master/Slave Replication ...
db2
Replicator
db1
Replicator Service db1
Service db1
db3
Replicator
Service db1
./cooobook/install_master_slave.sh
©Continuent 2013 16

17. How Do I Install Fan-In Replication?
db1
Replicator
Service db1
Replicator
Service db1
Service db2
db2
Replicator
Service db2 db3
./cooobook/install_fan_in.sh
©Continuent 2013 17

18. How Do I Install Multi-Master?
db1 Replicator
Service db1
Service db2
db2
Service db1
Service db2
Replicator
./cooobook/install_all_masters.sh
©Continuent 2013 18

19. How Do I Extend Multi-Master?
db1 Replicator
Service db1
Service db2
Replicator db3
Service db3
Service db1
Service db2
db2
Service db3
Service db1
Service db2
Service db3
Replicator
©Continuent 2013 19

20. How Do I Extend Multi-Master?
Replicator Replicator
db1
db3
Service db1 Service db1
Service db2 Service db2
Service db3 Service db3
Service db4 Service db4
db2
db4
Service db1 Service db1
Service db2 Service db2
Service db3 Service db3
Service db4 Service db4
Replicator Replicator
©Continuent 2013 20

21. How Do I Install a Star Topology?
db1
Replicator
Service db1
Replicator db3
Service db3
Service db1 HUB
Service db2
db2
Service db3
Service db2
Service db3
Replicator
./cooobook/install_star.sh
©Continuent 2013 21

22. How Do I Extend a Star Topology?
Replicator
db1
Service db1
db3
Service db1
Service db3
Service db2 HUB
Service db3
Replicator
db2 Service db4
Service db2
Service db3
db4
Service db3
Service db4
Replicator
©Continuent 2013 22

23. How Do I Extend a Star Topology?
Replicator
db1
Service db1
db3
Service db1
Service db3
Service db2 HUB
Service db3
Replicator
db2 Service db4
Service db2 Service db5
Service db3
Replicator
db4
db5
Service db5 Service db3
Service db3 Service db4
Replicator
©Continuent 2013 23

24. Overview of Tungsten
Cookbook
©Continuent 2012. 24

25. Tungsten Cookbook
tungsten-replicator-2.0.7-278
|
+--/cluster-home
+--/cookbook
+--/tools
+--/tungsten-replicator
©Continuent 2013 25

26. Tungsten Cookbook
tungsten-replicator-2.0.7-278
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_MASTER_SLAVE.sh
+--show_master_slave.sh
+--test_master_slave.sh
+--clear_cluster_master_slave.sh
...
©Continuent 2013 26

27. Tungsten Cookbook
tungsten-replicator-2.0.7-278
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_STAR.sh
+--show_star.sh
+--test_star.sh
+--clear_cluster_star.sh
...
©Continuent 2013 27

28. Tungsten Cookbook
tungsten-replicator-2.0.7-278
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_ALL_MASTERS.sh
+--show_all_masters.sh
+--test_all_masters.sh
+--clear_cluster_all_masters.sh
...
©Continuent 2013 28

29. Tungsten Cookbook
tungsten-replicator-2.0.7-278
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_FAN_IN.sh
+--show_fan_in.sh
+--test_fan_in.sh
+--clear_cluster_fan_in.sh
...
©Continuent 2013 29

30. Tungsten Cookbook
$ cat COMMON_NODES.sh
export NODE1=host1
export NODE2=host2
export NODE3=host3
export NODE4=host4
©Continuent 2013 30

31. Tungsten Cookbook
$ cat USER_VALUES.sh
# User defined values for the cluster to be
installed.
export TUNGSTEN_BASE=$HOME/installs/cookbook
export DATABASE_USER=tungsten
export BINLOG_DIRECTORY=/var/lib/mysql
export MY_CNF=/etc/my.cnf
export DATABASE_PASSWORD=secret
export DATABASE_PORT=3306
export TUNGSTEN_SERVICE=cookbook
export RMI_PORT=10000
export THL_PORT=2112
export START_OPTION=start
©Continuent 2013 31

32. Sample Star Installation
• edit cookbook/COMMON_NODES.sh
• edit cookbook/USER_VALUES.sh
• run cookbook/install_star.sh
• and then:
• run cookbook/show_star.sh
• run cookbook/test_star.sh
©Continuent 2013 32

33. CONFLICTS
©Continuent 2013 33

34. Con#ict Handling Strategies
• resolving planned for
future
• after the fact
• Needs information that is missing in async replication
• avoiding
• requires synchronous replication with 2pc
• preventing available now
• setting and enforcing a split sources policy
• transforming and resolving planned for
• all records are converted to INSERTs future
• con"icts are resolved within a given time window
©Continuent 2013 34

35. Multi-Master:
Con#ict Prevention
©Continuent 2012. 35

36. Tungsten Con#ict Prevention Facts
• Sharded by database
• De"ned dynamically
• Applied on the slave services
• methods:
• error: make replication fail
• drop: drop silently
• warn: drop with warning
©Continuent 2013 36

37. Tungsten Con#ict Prevention
Applicability
• unknown shards
• The schema being updated is not planned
• actions: accept, drop, warn, error
• unwanted shards
• the schema is updated from the wrong master
• actions: accept, drop, warn, error
• whitelisted shards
• can be updated by any master
©Continuent 2013 37

38. Con#ict Prevention Directives
--svc-extractor-filters=shardfilter
replicator.filter.shardfilter.unknownShardPolicy=error
replicator.filter.shardfilter.unwantedShardPolicy=error
replicator.filter.shardfilter.enforceHomes=false
replicator.filter.shardfilter.allowWhitelisted=false
©Continuent 2013 38

39. Con#ict Prevention in a Star Topology
Host1
A master: alpha
C database: employees
A Host3
master: charlie (hub)
B C
database: vehicles
Host2
master: bravo
database: buildings B C
©Continuent 2013 39

40. Setting Con#ict Prevention Rules
trepctl -host host1 -service charlie
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 1
©Continuent 2013 40

41. Sample Correct Operation (1)
mysql #1> create table employees.names( ... )
# all servers receive the table
# all servers keep working well
©Continuent 2013 41

42. Sample Correct Operation (2)
mysql #2> create table buildings.homes( ... )
# all servers receive the table
# all servers keep working well
©Continuent 2013 42

43. Sample Incorrect Operation (1)
mysql #2> create table employees.nicknames( ... )
# Only server #2 receives the table
# slave service in hub gets an error
# slave service in #1 does not receive anything
©Continuent 2013 43

44. Sample Incorrect Operation (2)
#3 $ trepct services | simple_services
alpha [slave]
seqno: 7 - latency: 0.136 - ONLINE
bravo [slave]
seqno: -1 - latency: -1.000 - OFFLINE:ERROR
charlie [master]
seqno: 66 - latency: 0.440 - ONLINE
©Continuent 2013 44

45. Sample Incorrect Operation (3)
#3 $ trepct -service bravo status
NAME VALUE
---- -----
appliedLastEventId : NONE
appliedLastSeqno : -1
appliedLatency : -1.0
(...)
offlineRequests : NONE
pendingError : Stage task failed: q-to-dbms
pendingErrorCode : NONE
pendingErrorEventId : mysql-bin.000002:0000000000001241;0
pendingErrorSeqno : 7
pendingExceptionMessage: Rejected event from unknown shard:
seqno=7 shard ID=employees
(...)
©Continuent 2013 45

46. Fixing the Issue
mysql #1> drop table if exists employees.nicknames;
mysql #1> create table if exists employees.nicknames ( ... ) ;
#3 $ trepct -service bravo online -skip-seqno 7
# all servers receive the new table
©Continuent 2013 46

47. Joining the Community
©Continuent 2013 47

48. Tungsten Replicator is Open Source
• Project home:
http://code.google.com/p/tungsten-replicator/
• Log bugs, "nd builds, post in replicator discussion
group
• Documentation:
https://docs.continuent.com/wiki/display/TEDOC/
Tungsten+Documentation+Home
©Continuent 2013 48

49. Wrapping Up
©Continuent 2013 49

50. Tungsten University Sessions
• Zero-Downtime Maintenance And Upgrade
- Thursday March 21st
- Tuesday March 26th
• Geographically Distributed Tungsten Clusters
- Thursday April 11th
- Tuesday April 16th
• Replicate Between MySQL And Oracle
- Thursday May 2nd
- Tuesday May 7th
• Send any feedback to: tu@continuent.com
©Continuent 2013 50

51. 560 S. Winchester Blvd., Suite 500 Our Blogs:
San Jose, CA 95128 http://scale-out-blog.blogspot.com
Tel +1 (866) 998-3642 http://datacharmer.org/blog
Fax +1 (408) 668-1009 http://www.continuent.com/news/blogs
e-mail: sales@continuent.com
Continuent Web Page:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
©Continuent 2012.