Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering

Skalbarhet och
tillgänglighet med
MySQL-replikering
Sven Sandberg (sven.sandberg@oracle.com)
Principal Software Engineer, MySQL Replication Core Team
(För tillfället föräldraledig)

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. |`2
Safe Harbour Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into any
contract.
It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The development,
release, and timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

Overview
MySQL Replication
 Design
 Setting Up

Industry Leaders Rely On MySQL
Replication  Design  Setting Up
Cloud
OEM & ISVsWeb & Enterprise

Replication: Copy Changes Master → Slave
 MySQL Master Server
– Changes data
– Sends changes to slave
 MySQL Slave Server
– Receives changes from master
– Applies received changes to database
M S

M M/S S
S
S
S
M
S
M
M
Server can be master, slave or both
Master can have multiple slaves
Slave can only have one master

M/S
S
M
M
Circular replication is also possible
You can simulate multiple
masters by time-sharing
M/S
M/S
M/S
M/S
M/S
S
M
M
Switch
every 5 sec

Why Replication? – Performance
 Read scale-out
M S
write clients read clients

 Read scale-out
M S
More
reads?
More
slaves!

 Read scale-out
M S
S
S
S
M
read clients
write clients
More
reads?
More
slaves!

Why Replication? – Redundancy
 If master crashes, promote slave to master
C
B
A

C
B
A
Crash

C
B
A
B is the
new master

Why Replication? – Long-distance Data
Distribution
CB
BAAC
Image from
www.ginkgomaps.com

Cool!
So, give me a 7 min
intro to replication.

Overview
MySQL Replication
 Design
 Setting Up

All Changes Written to Binary Log
binary log
Client

binary log
Client
create table t (a int);

create...
binary log
Client
Table t

create...
binary log
Client
Table t
insert into t values (1);

create...
insert...
binary log
Client
Table t
1
insert into t values (1);

Slave Initiates Replication
B
binary log
A
binary log
Client
1. Slave sends
request to start replication
to master
2. Master sends
stream of replication data
to slave

create...
Binary Log Sent to Slave, Re-executed
B
binary log
A
binary log
Client

create...
Table t
B
binary log
create...
A
binary log
Client

create...
Table t Table t
create...
B
binary log
create...
A
binary log
Client

create...
insert...
Table t Table t
create...
B
binary log
create...
A
binary log
Client

create...
insert...
Table t
1
Table t
create...
B
binary log
create...
insert...A
binary log
Client

create...
insert...
Table t
1
Table t
1
create...
insert...B
binary log
create...
insert...A
binary log
Client

Actually, Changes Land in Slave's Relay Log
B
binary logrelay log
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Network

create...
B
binary logrelay log
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Network

create...
B
binary logrelay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t
Network

create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t
Network

create...
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Network

create...
insert...
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Network

create...
insert...
create...
B
binary log
create...
relay log
create...
insert...A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t
1
Table t
Network

create...
insert...
create...
B
binary log
create...
insert...
relay log
create...
insert...A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t
1
Table t
Network

create...
insert...
create...
insert...B
binary log
create...
insert...
relay log
create...
insert...A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t
1
Table t
1
Network

create...
Replication is Asynchronous
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Atomic → db and log
consistent after crash
Asynchronous Asynchronous Asynchronous
(Atomic
too)
(Atomic
too)
Network

Replication
seems fantastic!
I want to try!

Overview
MySQL Replication
 Design
 Setting Up

Setting Up Replication
 Master's my.cnf:
serverid=1
logbin=masterbin
 Slave's my.cnf:
serverid=2
logbin=slavebin
relaylog=slaverelaybin
logslaveupdates

Setting Up Replication
 On master: create replication user
> CREATE USER `rpl_user`@`slave_host`;
> GRANT REPLICATION SLAVE ON *.*
    TO `rpl_user`@`slave_host` IDENTIFIED BY 'secret';
> FLUSH PRIVILEGES;
 On slave: initiate replication
> CHANGE MASTER TO
  MASTER_HOST = 'master_host', MASTER_PORT = <port>,
  MASTER_USER = 'rpl_user', MASTER_PASSWORD = 'secret';
> START SLAVE;

Replicate!
 On master: execute some SQL
> USE test_db;
> CREATE TABLE t1 (a INT);
> INSERT INTO t1 VALUES (1);
 On slave: check that it replicated:
> SELECT * FROM test_db.t1;
a
1

Check Replication Status
 On master:
> SHOW BINARY LOGS;
> SHOW BINLOG EVENTS;
Log_name File_size
master-bin.000001 426
Log_name Pos Event_type sid end_pos Info
master-bin.000001 4 Format_desc 1 120 Server ver: 5.6.6-m9-log, Binlog ver: 4
master-bin.000001 120 Query 1 217 use `test`; CREATE TABLE t1 (a INT)
master-bin.000001 217 Query 1 296 BEGIN
master-bin.000001 296 Query 1 395 use `test`; INSERT INTO t1 VALUES (1)
master-bin.000001 395 Xid 1 426 COMMIT /* xid=21 */
One binary log
so far
Five events:
One CREATE and
one INSERT

Check Replication Status
 On slave:
> SHOW SLAVE STATUSG
…
Slave_IO_Running Yes
Slave_SQL_Running Yes
…
Last_IO_Errno 0
Last_IO_Error
Last_SQL_Errno 0
Last_SQL_Error
…
Both slave threads
are running
If any thread stops
with an error,
look for error
message here

Next? Vote!
1. Fail-over + Crash tolerance
2. Fail-over + Q&A
3. Q&A

Overview
MySQL Replication
Crash-Tolerance
 Failover
 Reduce Risk for Data Loss
– Semi-sync
– Transactional meta-data
 Performance
– Multi-threaded slave

Crash-Tolerance
 Can happen to anyone:
– Hardware failure
– Human mistake
– Bug
– Natural disaster
 Features in MySQL 5.6:
– Reduce admin overhead
– Prevent unplanned downtime
– Prevent planned downtime
Crash-Tolerance  Fail-over  Reduce Risk for Data Loss  Performance

Crash-Tolerance
 Fail-over: “back on track” faster & more automatically
– Global Transaction Identifiers
 Reduce risk for data loss: less administration → less downtime
– Semi-synchronous replication → slave has all changes from master
– Update replication metadata transactionally → safe on disk
 Performance: help slave keep up with master
– Replication metadata in table – leverage disk sync from storage engine
5.5
5.6

OK.
Tell me about
fail-over!

Overview
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Fail-over
 Handle server crash with minimal disruption
 Example 1: tree
C
B
A
Crash-tolerance  Fail-over  Reduce Risk for Data Loss  Performance

Fail-over
 Example 1: tree
C
B
A
Crash

Fail-over
C
B
A
 Example 1: tree

Fail-over
C
B
A
Make B the new master
 Example 1: tree

Fail-over
CA B
 Example 1: tree
 Example 2: line

Fail-over
CA B
 Example 1: tree
 Example 2: line
Crash

Fail-over
CA B
 Example 1: tree
 Example 2: line

Fail-over
CA B
Make A direct master of C
 Example 1: tree
 Example 2: line

Fail-over
C
A
B
D
 Example 1: tree
 Example 2: line
 Example 3: circle

Fail-over
C
A
B
D
 Example 1: tree
 Example 2: line
Crash

Fail-over
C
A
B
D
 Example 1: tree
 Example 2: line

Fail-over
C
A
Make a shortcut
in the circle
B
D
 Example 1: tree
 Example 2: line

Fail-over
C
A
B
D
 Example 1: tree
 Example 2: line
 Example 4: scheduled maintenance

Fail-over
C
A
B
D
Take
out D!
 Example 1: tree
 Example 2: line

Fail-over
C
A
B
D
Maintain
D offline,
without
disrupting
service
 Example 1: tree
 Example 2: line

Fail-over
C
A
B
D
 Example 1: tree
 Example 2: line

Fail-over
 Example 1: tree
 Example 2: line
 Example 5: arbitrary topology
A
B
C
D
E
F

Fail-over: Design
 Server assigns Global Transaction Identifier (GTID) to every
transaction:
– server_uuid:number
a61678ba488942799e5845ba840af334:1
– server_uuid identifies the server; globally unique
– number is incremented by 1 for each transaction on this server
 Writes GTID to binary log
 Slave preserves GTID when re-executing transaction

Fail-over: Design: New Events In Binary Log
> SHOW BINLOG EVENTS;
Log_name Pos Event_type sid end_pos Info
master-bin.000001 4 Format_desc 1 120 Server ver: 5.6.12-m9-log, Binlog ver: 4
master-bin.000001 120 Prev_gtids 1 151
master-bin.000001 151 Gtid 1 199 SET @@GTID_NEXT= '<uuid>:1'
master-bin.000001 199 Query 1 296 use `test`; CREATE TABLE t1 (a INT)
master-bin.000001 296 Gtid 1 344 SET @@GTID_NEXT= '<uuid>:2'
master-bin.000001 344 Query 1 423 BEGIN
master-bin.000001 423 Query 1 522 use `test`; INSERT INTO t1 VALUES (1)
master-bin.000001 522 Xid 1 602 COMMIT /* xid=21 */
3 new events:
1 Previous_gtids +
2 Gtid
file
header
trans-
action
trans-
action

Fail-over: Design: Protocol
 Master-slave protocol:
– Slave sends to master:
range of identifiers of executed transactions to master
– Master sends all other transactions to slave
(slave)
id1,trx1,
id2,trx2
(master)
id1,trx1,
id2,trx2,
id3,trx3
binlog
A 2. id3, trx3, …
1. id1…id2
binlog
B

 Example 1: tree
A
(crashed)(master)
A
id1,trx1,
id2,trx2,
id3,trx3
binlog
(slave)
C
id1,trx1
binlog
(slave)
id1,trx1,
id2,trx2B
binlog

 Example 1: tree
A
(crashed)(master)
A
id1,trx1,
id2,trx2,
id3,trx3
binlog
(slave)
C
id1,trx1
binlog
(slave)
id1,trx1,
id2,trx2B
binlog
Crash

 Example 1: tree
(slave)
C
id1,trx1
binlog
(slave)
id1,trx1,
id2,trx2B
binlog
A
(crashed)(crashed)
A
id1,trx1,
id2,trx2,
id3,trx3
binlog id1
id2, trx2,...

 Example 1: tree
B
binlog
A
binlog
C
binlog
client client

 Example 1: tree
id1,trx1
B
binlog
A
binlog
C
binlog
client client trx1

 Example 1: tree
id1,trx1,
id2,trx2B
binlog
id2,trx2
A
binlog
C
binlog
client clienttrx2 trx1

 Example 1: tree
id1,trx1,
id2,trx2,
id3,trx3
B
binlog
id2,trx2
A
binlog
C
binlog
client clienttrx2 trx1, trx3

 Example 1: tree
id1,trx1,
id2,trx2,
id3,trx3
B
binlog
id2,trx2
A
binlog
C
binlog
Crash

 Example 1: tree
id1,trx1,
id2,trx2,
id3,trx3
B
binlog
id2,trx2
A
binlog
C
binlog
id2
id1,trx1,id3,trx3,...

Wow!
Show me how to
configure for
fail-over!

Fail-over: Set Up
 Prerequisites:
– Configure replication as usual
– Use transactional storage engine for all tables (InnoDB)
– Don't use CREATE TABLE … SELECT

Fail-over: Set Up
 Prepare the server for fail-over
1. Sync and stop all servers
2. Add to every my.cnf:
gtidmode=on
enforcegtidconsistency=on
logbin
logslaveupdates
3. Start all servers
4. Create replication user on every server that may become master
5. Execute:
> CHANGE MASTER TO MASTER_AUTO_POSITION = 1

Fail-over: Perform
 Perform fail-over
On each slave:
> CHANGE MASTER TO MASTER_HOST = '<host>',
                   MASTER_PORT = <port number>,
                   MASTER_USER = '<user name>'
                   MASTER_PASSWORD = 'secret';
Credentials on
new master

Fail-over: Perform
C
B
A
CHANGE MASTER TO
MASTER_HOST = 'B',
MASTER_PORT = <B's port>,

Fail-over: Perform
CHANGE MASTER TO
MASTER_HOST = 'C',
MASTER_PORT = <C's port>,
C
A
B
D

I love GTIDs!
Can fail-over be
automated?

Fail-over: Utilities
 Mysqlrpladmin
– Promote one slave to master
– mysqlrpladmin switchover
  master='user[:pass]@host[:port][:sock]'
  discoverslaveslogin='user[:pass]'
  newmaster='user[:pass]@host[:port][:sock]'
– Many other admin tasks

 Mysqlfailover
– Monitor master
– If master goes away, do automatic fail-over
– mysqlfailover
master='user[:password]@host[:port[:socket]]'
discoverslaveslogin='user:password'

 Video tutorial
– dev.mysql.com/techresources/articles/mysql
replicationutilities.html
 Blog
– drcharlesbell.blogspot.co.uk/2012/04/mysql
utilitiesandglobaltransaction.html
 Doc
– dev.mysql.com/doc/workbench/en/mysqlfailover.html
– dev.mysql.com/doc/workbench/en/mysqlrpladmin.htm
 Download
– www.mysql.com/downloads/workbench/

Now I can
do fail-over!
But how do I avoid
losing data after
a crash?

Overview
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Overview - less manual work
- reduce downtime
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Overview - less manual work
- reduce downtime
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance
a.k.a. “slave has all
changes from master”

Asynchronous vs Semi-sync Replication
 By default, replication is asynchronous
– Master does not wait for slave
– Fast
– ACK to app before slave receives change → changes lost if master dies
 New in MySQL 5.5: semi-synchronous replication plugin
– Master acks to app
only after changes are in slave's relay log
– Master has to wait
– Less risk for lost updates

create...
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Network

create...
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Asynchronous Asynchronous Asynchronous
(Atomic
too)
(Atomic
too)
Network

create...
Semi-synchronous Replication
create...
B
binary log
create...
relay log
create...
A
binary log
Client
Dump
thread
IO
thread
SQL
thread
Table t Table t
Asynchronous
Network
Semi-synchronous replication:
statement is in slave's relay log
before client ack

Semi-synchronous Replication: Set Up
 Consists of two plugins: master plugin, slave plugin
 Installation: read the manual :-)
– http://dev.mysql.com/doc/refman/5.6/en/
replicationsemisyncinstallation.html
 After installation:
– On master:
SET GLOBAL rpl_semi_sync_master_enabled = {0|1};
– On slave:
SET GLOBAL rpl_semi_sync_slave_enabled = {0|1};

So semi-sync
ensures slave has all
changes before my app
gets notified.
Anything else that helps
consistency after
a crash?

Overview
Meta-data
safe on disk
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Replication Metadata in Tables
 Slave threads maintain their current positions
– IO thread: where to read from master
– SQL thread: where to read from relay log
 Positions stored on disk
– Before: in file
– 5.6: in table
insert...
B
binary log
insert...
relay log
IO
thread
SQL
thread

Replication Metadata in Tables
 File: updated after
transaction commit
Trans-
action
Update
position
Trans-
action
Update
position
Crash
possible
 Table: updated atomically at
transaction commit
 Consistent after server crash

Replication Metadata in Tables: Set Up
 Only meaningful if using transactional storage engine
 Add to slave's my.cnf:
masterinforepository=TABLE
slaveinforepository=TABLE
defaultstorageengine=InnoDB

Nice!
Also, won't slave be more
in sync if we improve
slave performance?

Overview
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance
Slave less behind
→ less lost updates,
short fail-over time

Overview
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance
Slave less behind
→ less lost updates,
short fail-over time
Fewer disk syncs

Performance: Replication Metadata In Tables
 File: separate disk sync
for the file
Trans-
action
Update
position
Trans-
action
Update
position
Sync
 Table: leverage sync from
storage engine
 (even better: group commit)
 Again, use transactional engine
Sync Sync

Ahh, so
transactional meta-data
is double-good!
How else can I improve
slave performance?

Overview
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Performance: Parallel Slave Threads
B
binary logrelay log
A
binary log
Client
Client
Parallel writer
threads on master
Writes are serialized
in binary log
Single applier
on slave →
Can become
bottleneck
Client

B
binary logrelay log
A
binary log
Client
ClientClient
Coordinator
thread (reading)
Multiple worker
threads! (executing)

 Parallelize by schema (database)
– Different worker threads never update same schema
– Effective if you partition data by schema
 Consistency
– Consistent within schema
– Eventually consistent between schemas
– Commit order may differ between master and slave, among slaves

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
relay log
3 worker
threads
Coordinator
thread
w1
w2
w3

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB1
w1
w2
w3
relay log

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB1
DB2
w1
w2
w3
relay log

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB1
DB2
w1
w2
w3
DB3
relay log

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB1
DB2
w1
w2
w3
DB3
Coordinator waits for w1 to finish
before feeding DB1 update to w3
relay log

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB2
w1
w2
w3
DB3
relay log

B
Trx1: DB1
Trx2: DB2
Trx3: DB3+DB1
3 worker
threads
Coordinator
thread
DB2
w1
w2
w3
DB3
DB1
relay log

Performance: Parallel Slave Threads: Set Up
 On slave:
> STOP SLAVE;
> SET SLAVE_PARALLEL_WORKERS = 4;
> START SLAVE;
Depends on number of CPUs
and number of databases.
No exact formula –
measure what's best for you!

Thank you,
that's a lot of info!
Can you
summarize?

Summary
MySQL Replication
Crash-Tolerance
 Failover
– Semi-sync
 Performance

Semi-sync
Global
Transaction
Identifiers
Replication
Base
Transactional
Metadata
Parallel slave
 My.cnf:
serverid=<different number for each server>
logbin=<host>bin
relaylog=<host>relaybin
logslaveupdates
masterinforepository=TABLE
slaveinforepository=TABLE
defaultstorageengine=InnoDB
gtidmode=on
enforcegtidconsistency=on
 Client commands:
> CHANGE MASTER TO MASTER_AUTO_POSITION = 1;
> SET SLAVE_PARALLEL_WORKERS = 4;
> SET GLOBAL rpl_semi_sync_master_enabled = {0|1};
> SET GLOBAL rpl_semi_sync_slave_enabled = {0|1};
Summary

Thank you!
Questions?Email: sven.sandberg@oracle.com
Blog: svenmysql.blogspot.co.uk

Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering

Recomendados

Recomendados

Más contenido relacionado

Similar a Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering

Similar a Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering (20)

Último

Último (20)

Drupal Camp Göteborg 2013: Skalbarhet och tillgänglighet med MySQL-replikering