Más contenido relacionado La actualidad más candente (20) Similar a Geographically Distributed Multi-Master MySQL Clusters (20) Geographically Distributed Multi-Master MySQL Clusters2. ©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
3. ©Continuent 2014
Quick Continuent Facts
• Largest Tungsten clustering installation
processes up to 700M transactions daily
• Over 100 customers including many
household names
• Multiple multi-site, multi-master
deployments in production
• ... And more are on the way
3
5. ©Continuent 2014
Tungsten Database-as-a-Service
5
Tungsten clusters combine off-
the-shelf open source DBMS
servers into data services with:
!
• 24x7 data access
• Scaling of load on replicas
• Simple management commands
!
...without app changes or data
migration
Amazon
US West
apache
/php
GonzoPortal.com
Connector Connector
6. ©Continuent 2014
Cluster Detailed View
6
Application
Tungsten Connector
Application
Tungsten Connector
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
Manager Manager Manager
MonitoringandControl
MonitoringandControl
Data Service: sjc
8. ©Continuent 2014
Tungsten Replicator Overview
8
Master
(Transactions + Metadata)
Slave
THL
DBMS
Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download
transactions
via network
Apply using JDBC
11. ©Continuent 2014
Using Services for Multi-Master
Replication
11
New York London
Replicator
lhr (master)
jfk (slave)
Replicator
lhr (slave)
jfk (master)
13. ©Continuent 2014
Why Multi-Site? Why Multi-Master?
13
• Get data close to global applications
• Avoid latency cost of commit between sites
• Keep running during intermittent network
failures between sites
• Keep running if a site fails
• Enable simple recovery when it comes back
14. ©Continuent 2014
Multi-Site Multi-Master Clustering:
the Marketecture
14
No app or
data coupling
across regions
ApplicationApplications ApplicationApplications
Amazon Region Amazon Region
Live data
and apps
in all regions
Multi-AZ
deployment
Async replication
between sites
Run in Amazon
or on-premises
17. ©Continuent 2014
Puppet Installation and Apply
17
$> puppet module install continuent/tungsten
$> puppet apply ~/tungsten.pp
See https://github.com/continuent/
continuent-vagrant for more examples
19. ©Continuent 2014
Best Practice Deployment in Amazon
19
Multi-AZ clustering with asynchronous multi-
master across regions
US-East-1
AZAZ
AZ AZ
EU-
West-1
AZ
AZ AZ
US-
West-1
AZ
AZ AZ AZAZ AZ AZAZ
Availability ZoneRegion
AZ AZ AZ
26. ©Continuent 2014
What Are Replication Filters?
26
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Master Pipeline
Master
DBMS
Transaction
History Log
In-Memory
Queue
Inspect
Change
Drop
27. ©Continuent 2014
Problem: Handling PII Data
27
CREATE TABLE `personal_info` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`userid` varchar(250) DEFAULT NULL,
`personal_code` varchar(250) DEFAULT NULL,
`birth_date` date DEFAULT NULL,
`email` varchar(250) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Requirement 1: Do not replicate
personally identifiable information
28. ©Continuent 2014
Problem: Application-Specific Rules
28
mysql> select * from personal_info;
+----+--------+---------------+------------+-----------------+
| id | userid | personal_code | birth_date | email |
+----+--------+---------------+------------+-----------------+
| 1 | fre11 | secret | 1985-01-16 | bobr@gmail.com |
| 2 | bill | verysecret | 1995-01-16 | billk@gmail.com |
+----+--------+---------------+------------+-----------------+
Requirement 2: prevent userid
‘bill’ from replicating
29. ©Continuent 2014
Adding Filters to Meet Requirements
29
Extract Filter(s) Apply
Stage
Extract Filter Apply
Stage
Master Pipeline
Master
DBMS
Transaction
History Log
In-Memory
Queue
colnames
killbill
dropcolumns
dropcolumns.json
30. ©Continuent 2014
Creating a dropcolumn.json File
[
{
"schema": "test",
"table": "personal_info",
"columns": [
"personal_code",
"birth_date",
"email"
]
}
]
30
Table name
Columns to delete
31. ©Continuent 2014
Writing a Custom killbill.js Filter
function filter(event) {!
data = event.getData();!
if(data != null) {!
for (i = 0; i < data.size(); i++) {!
d = data.get(i);!
if(d != null && d instanceof com.continuent.tungsten.replicator.dbms!
.RowChangeData) {!
rowChanges = d.getRowChanges();!
for(j = 0; j < rowChanges.size(); j++) {!
oneRowChange = rowChanges.get(j);!
schema = oneRowChange.getSchemaName();!
table = oneRowChange.getTableName();!
if (schema == "test" && table == "personal_info") {!
columnValues = oneRowChange.getColumnValues();!
for (row = 0; row < columnValues.size(); row++) {!
raw = columnValues.get(row).get(1).getValue();!
user = new java.lang.String(raw);!
if (user.equals("bill")) {!
logger.info("Dropping transaction from bill!");!
return null;!
} } } } } } } }!
31
Drop transaction
32. ©Continuent 2014
Killbill.tpl file for Filter Installation
# Drop transactions for bill.
replicator.filter.killbill=com.continuent.tungsten.replicator.
filter.JavaScriptF
ilter
replicator.filter.killbill.script=/opt/continuent/share/
killbill.js
32
33. ©Continuent 2014
Installing the Filters...
cp dropcolumn.json /opt/continuent/share
cp killbill.js /opt/continuent/share
cp killbill.tpl /opt/tungsten/tungsten-replicator/samples/
conf/filters/default
!
/opt/continuent/tungsten/tools/tpm update
--svc-extractor-filters=colnames,killbill,dropcolumn
--property=replicator.filter.dropcolumn.definitionsFile=/
opt/continuent/share/dropcolumn.json
--start-and-report=true
33
34. ©Continuent 2014
Resulting Data on Slave(s)
34
mysql> select * from personal_info;
+----+--------+---------------+------------+-------+
| id | userid | personal_code | birth_date | email |
+----+--------+---------------+------------+-------+
| 1 | fre11 | NULL | NULL | NULL |
+----+--------+---------------+------------+-------+
1 row in set (0.00 sec)
!
Requirement 2: No bill!
Requirement 1: PII columns cleared
36. ©Continuent 2014
How Do I Get Started?
36
• Documentation for all products
http://docs.continuent.com
• Download registration for Tungsten clustering:
http://www.continuent.com/downloads/software
• Downloads for open source Tungsten Replicator
http://code.google.com/p/tungsten-replicator/
• Vagrant project for setting up test topologies
https://github.com/continuent/continuent-vagrant/tree/master/
examples/MSMM
37. ©Continuent 2014
Continuent Web Page:
http://www.continuent.com
!
Tungsten Replicator:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.org/blog
http://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com