Do you need run multi-master MySQL database clusters across sites? In this webinar we build on the multi-master capabilities of Continuent Tungsten, covered in our advanced replication topologies course, to help you build and manage systems that spread data across dozens or even hundreds of sites. We cover topics such as large scale topologies, transaction filtering, security, and moving replication from simple data movement to application messaging. Course Topics: - Review of multi-master replication topologies - Setting up Continuent Tungsten clusters for local site HA - Linking Continuent Tungsten clusters across sites - Security for cross-site replication - Handling local as well as site-wide failures - Off-the-shelf filters to improve multi-master replication - Custom filter development to implement business rules for application messaging.

1. Tungsten University:
Geographically Distributed
Multi-Master MySQL Clusters
Robert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013

2. Introducing Continuent
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
©Continuent 2013 2

3. Geographic Distribution of Data
©Continuent 2013 3

4. Why Multi-Master Across Sites?
• Move applications close to customers
• Online game pro"les
• Mapping app information
• Protect against site failures
• Hosted call centers
• Credit card processing
Multi-master is NOT a performance scaling solution
©Continuent 2013 4

5. Tungsten Cluster Architecture
in 3 Minutes or Less
©Continuent 2013 5

6. Overview of a Tungsten Cluster
Application Application
Tungsten Connector Tungsten Connector
Monitoring and Control
Monitoring and Control
Db2 Db1 Db3
Manager Manager Manager
Replicator Replicator Replicator
Slave Master Slave
Data Service: sjc
©Continuent 2013 6

7. Tungsten Replicator Architecture
in 2 Minutes or Less
©Continuent 2013 7

8. Tungsten Replicator Overview
Master
Replicator THL
Download
transactions
via network (Transactions + Metadata)
DBMS
Logs
Slave Replicator THL
Apply using JDBC (Transactions + Metadata)
©Continuent 2013 8

9. Simple Multi-Master Con"guration
London Replicator New York
Replicator
ny (slave) ny (master)
lon (master) lon (slave)
Multiple services per replicator
©Continuent 2013 9

10. Cross-Site Clustering in Action
©Continuent 2013 10

11. Clustered Multi-Master Con"guration
London New York
te1 master ny lon master te3
ny lon
te2 slave slave te4
Cross-site replicators Cross-site updates
read from slaves are unlogged
©Continuent 2013 11

12. Reconnect to Slave after Switch
London New York
te1 slave ny lon master te3
ny lon
te2 master slave te4
(Master switch to te2)
©Continuent 2013 12

13. Connect to Master when Slave Fails
London New York
te1 slave ny lon master te3
ny lon
te2 master slave te4
(Slave failure/maintenance)
©Continuent 2013 13

14. Multi-Master Best Practices
©Continuent 2013 14

15. Eventually Consistent Design
• Each site is independent application with
local HA cluster
• Replication links clusters
• Transfer is asynchronous
• Think through business implications
• What does it mean if a site is down for several days?
• How do I get up-to-date picture of data?
©Continuent 2013 15

16. Avoid Con#icting Primary Keys
• Method #1: Auto-increment o$sets
[my.cnf]
server-id=1
auto-increment-offset = 1
auto-increment-increment = 4
• Method #2:
create table test.keys (
server_id int primary key,
counter int);
insert into test.keys(server_id, counter)
values(@@server_id, 0);
©Continuent 2013 16

17. Use Row Replication
• Row replication sends changes to each row:
[my.cnf]
binlog_format=row
• Statement replication results in ambiguous
changes depending on state of replication
update employees
set salary = salary * 1.1;
Can affect different numbers of
employees depending on state of
replication between sites
©Continuent 2013 17

18. Watch Out for...
• Triggers -- Tungsten cannot turn them o$;
you have to alter the code
• Scheduled Events -- Be careful where they
run!
• MyISAM -- Not crash safe and just generally
bad for replication
• Very large transactions --
©Continuent 2013 18

19. Look Out for Con#icts
• UPDATE/DELETE on same rows across
di$erent sites
• Generation of globally unbroken sequences
(e.g., invoice numbers)
• Reports that need global state from all sites
©Continuent 2013 19

20. Use Filters to Detect/Avoid Con#icts
Name Purpose
ignore_server Drop transactions from server-id
rename Rename schemas/tables/columns
replicate Control schemas/table replication
shardfilter Control shard replication
You can also write your own filters!
©Continuent 2013 20

21. Test, Test, Test!!!
• Set up a QA environment
• Put load on every master on every site
• Test backup and restore on every DBMS node
• Test switch
• Test failover
• Compare data between sites
• Develop a reset procedure for fast test cycles
©Continuent 2013 21

22. Be Ready For Trouble
• Ensure you have multiple days of logs so you
can be o%ine for a while
• Learn how to use data repair tools like pt-
table-checksum
• Remove constraints to make repair easier
• Monitor replication!
• Get a support contract for when things go
wrong
©Continuent 2013 22

23. Advanced Multi-Site Clustering
©Continuent 2013 23

24. Complex Multi-Master Topologies
All
Masters
Star
Replication
©Continuent 2013 24

25. Snow#ake Topology
HQ
Region 1 Region 2
CM1 CM2 CM1 CM2
©Continuent 2013 25

26. Wrapping Up
©Continuent 2013 26

27. Tungsten University Sessions
• Replicate between MySQL and Oracle
(May 2 and 7)
Send any feedback to: tu@continuent.com
©Continuent 2013 27

28. 560 S. Winchester Blvd., Suite 500 Our Blogs:
San Jose, CA 95128 http://scale-out-blog.blogspot.com
Tel +1 (866) 998-3642 http://datacharmer.org/blog
Fax +1 (408) 668-1009 http://www.continuent.com/news/blogs
e-mail: sales@continuent.com
Continuent Web Page:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
©Continuent 2012.