MySQL 5.7 MTS (Multi Thread Slave)

MySQL5.7
- replication
LEE SAE WOONG
2016.03.26

INDEX
> Multi-Threaded Slave(MTS)
2

Multi-Threaded Slave 3
binary log
Client
Client
Client
MASTER relay log SLAVE
* Before MySQL 5.6 : Single-threaded slave
Bottleneck

binary log
Client
Client
Client
MASTER relay log SLAVE
binary log
Client
Client
Client
MASTER relay log
* Before MySQL 5.6 : Single-threaded slave
Bottleneck
* MySQL 5.6 : Multi-threaded slave by database
db1
SLAVE
db2
db3

> Multiple execution threads to apply replication events to the slave(s);
> Splits processing between worker threads based on :
- Schema(MySQL 5.6+)
ㆍSET GLOBAL slave_parallel_type = DATABASE;
ㆍMulti-tenant systems friendly;
- Locking-based parallelism(new in MySQL 5.7.2)
ㆍSET GLOBAL slave_parallel_type = LOGICAL_CLOCK;
ㆍIntra-schema parallelization;
ㆍBased on information collected during master’s commit of the
transactons;
> DATABASE : parallelize by database
> LOGICAL_CLOCK : parallelize by master concurrency

6
binary log
Client
Client
Client
MASTER relay log
* MySQL 5.7.2 : Multi-threaded slave by master concurrency
Multi-Threaded Slave
Receiver
thread
relay log
COORDINATOR
Thread db1
Reads &
assigns to worker(s)
Applier
threads
worker1
worker2
worker3
SLAVE
SLAVE
db1

7Multi-Threaded Slave
Configure MTS
> STOP SLAVE;
> SET GLOBAL slave_parallel_type='LOGICAL_CLOCK';
> SET GLOBAL slave_parallel_workers=5;
> START SLAVE;

8
Receiver thread(a.k.a. IO thread)
Coordinator thread
(a.k.a SQL thread)
Applier threads
Show processlist for MTS
ㆍ Waiting for the next event in relay log :
The initial state before Reading event from the relay log.
ㆍ Reading event from the relay log :
The thread has read an event from the relay log so that the event can be processed.
ㆍ Making temp file :
The thread is executing a LOAD DATA INFILE statement and is creating a temporary file
containing the data from which the slave will read rows.
ㆍ Slave has read all relay log; waiting for the slave I/O thread to update it :
The thread has processed all events in the relay log files, and is now waiting for the
I/O thread to write new events to the relay log.
ㆍ Waiting for slave mutex on exit :
A very brief state that occurs as the thread is stopping.
https://dev.mysql.com/doc/refman/5.5/en/slave-sql-thread-states.html
http://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html

9
Replication Monitoring : P_S Replication Tables
performance_schema > show tables;
+----------------------------------------------------------+
…… |
…… |
| replication_applier_configuration |
| replication_applier_status |
| replication_applier_status_by_coordinator |
| replication_applier_status_by_worker |
| replication_connection_configuration |
| replication_connection_status |
| replication_group_member_stats |
| replication_group_members |
…… |
…… |
+----------------------------------------------------------+
https://dev.mysql.com/doc/mysql-perfschema-excerpt/5.7/en/performance-schema-replication-tables.html

10
> replication_applier_status_by_coordinator
> replication_applier_status_by_worker
The REPLICATION P_S Tables
SHOW SLAVE STATUS
> Do not have information about MTS workers;

Use case
Case1.
T1. update T set col1 = 2 where col1=3;
Case2.
x=1, y=1, z=99
On master we apply T1 and T2 in that order
T1. Update T set x=Read(y)+1 where z = 99;
T2. Update T set y=Read(x)+1 where z = 99;
On the slave however these two transactions commit out out order

Use case
Case1.
Case2.
x=1, y=1, z=99
On master we apply T1 and T2 in that order
On the slave however these two transactions commit out out order
x=2 , y=3
X=3, y=2

> slave_parallel_type= LOGICAL_CLOCK
•Leverage parallelization information obtained from the execution on the master:
–Transactions that prepare on the same “version” of the database, are assigned
the same timestamp;
같은 프리페어 트랜잭션은 같은 시간으로 어사인된다.
> parallelization conditions :
Same commit_parent id ( prepare stage)
> The pseudo code is as follows.
During Prepare
trx.commit_parent= commit_clock.get_timestamp();
During Commit
for every binlog group
commit_clock.step();
Logical clock and commit parent

Logical clock and commit parent
http://geek.rohitkalhans.com/2013/09/enhancedMTS-deepdive.html

ORACLE – 2013.11.13 ORACLE – 2014.09.23
OOW - 2015

transaction_info.h
/*Binlog-specific logical timestamps.*/
/*
Store for the transaction's commit parent sequence_number.
The value specifies this transaction dependency with a"parent"
transaction.
The member is assigned, when the transaction is about to commit
in binlog to a value of the last committed transaction's
sequence_number.
This and last_committed as numbers are kept ever incremented
regardless of binary logs being rotated or when transaction
is logged in multiple pieces.
However the logger to the binary log may convert them
according to its specification.
*/
int64 last_committed;
/*
The transaction's private logical timestamp assigned at the
transaction prepare phase. The timestamp enumerates transactions
in the binary log. The value is gained through incrementing
(stepping)a global clock.
Eventually the value is considered to increase
max_committed_transaction
system clock when the transaction has committed.
*/
int64 sequence_number;

17
Type slave-parallel-type slave_parallel_workers Syntax
master_info_repository
/ relay_log_info_repository
Master - - - FILE/TABLE
Slave 1 LOGICAL_CLOCK 16 (MTS)
STOP SLAVE;
SET GLOBAL slave_parallel_type='LOGICAL_CLOCK';
SET GLOBAL slave_parallel_workers=16;
START SLAVE;
FILE/TABLE
Slave 2 DATABASE 16 (MTS)
STOP SLAVE;
START SLAVE;
FILE/TABLE
Slave 3 - 0 (single thread) - FILE/TABLE
> master_info_repository – introduced 5.6.2 :
The setting of this variable determines whether the slave logs master status and connec
tion information to a FILE (master.info), or to a TABLE (mysql.slave_master_info).
> relay_log_info_repository – introduced 5.6.2 :
This variable determines whether the slave's position in the relay logs is written to a FIL
E (relay-log.info) or to a TABLE (mysql.slave_relay_log_info).
> Changing slave repository :
stop slave;
set global master_info_repository = 'table';
set global relay_log_info_repository = 'table';
start slave;
LOAD TEST
> CPU:2sx4c HT , Mem:32G, Disk:SAS 15Krpm 300G 4EA, Raid1+0 , 1 Master-3 Slaves

18
Master - - - FILE/TABLE
STOP SLAVE;
START SLAVE;
FILE/TABLE
STOP SLAVE;
START SLAVE;
FILE/TABLE
sysbench --test='/usr/share/doc/sysbench/tests/db/oltp.lua' --oltp_tables_count=1
--report-interval=10 --oltp-table-size=3000000 --num-threads=16 --mysql-user=root
--mysql-password=test --mysql-table-engine=innodb --rand-init=on
--mysql-socket=/mysql/MyHome/tmp/mysql.sock
--mysql-host=localhost --mysql-port=20306 --mysql-db=sysbench
prepare
mysqldump --skip-extended-insert --no-create-info -p'test' sysbench > dumpfile.sql
create database sysbench1;
create table sysbench1.sbtest1 like sysbench.sbtest1;
cat dumpfile.sql | sed -e's/`sbtest1`/sysbench1.sbtest1/g' | mysql -p'test' &
…
…
…
LOAD TEST

LOAD TEST
Thread1 : insert
MASTER
SLAVE 1
(logical_clock)
SLAVE 2
(database)
SLAVE 3
(single thread)
Thread16 : insert
Thread2 : insert
Thread3 : insert
●
●
●
●
●
●
●
Thread4 : insert
Thread5 : insert
Thread6 : insert

20
> Case #1 – 1 database
>> Performing simple insert query in 16 threads (1 database,FILE)
>>> slogical_clock을 설정했을 경우 parallel하게 처리되는 것을 확인 할 수 있다. (slave1)
>>> database가 1개 일때는 worker thread로 parallel 하게 처리가 되지 않는것을 확인 할 수 있다. (slave2)
>>> single thread가 오히려 slave2 보다 QPS가 약간 높게 나왔다. (slave3)
0
5000
10000
15000
20000
25000
master slave1 (logical_clock) slave2 (database) slave3 (single thread)
insertQPS
Case #1 – 1 database,FILE

21
> Case #2 – 4 databases
>> Performing simple insert query in 16 threads on masterDB (4 databases,FILE)
>>> database가 4개 일때 worker thread로 parallel 하게 처리되는 것을 확인 할 수 있다. (slave2)
>>> single thread 처리는 database가 1개 일때와 동일한 결과를 가져온다. (slave3)
0
5000
10000
15000
20000
25000
insertQPS

22
>> Performing simple insert query in 16 threads (1 database, TABLE)
>>> database가 1개 일때는 worker thread로 parallel 하게 처리가 되지 않는것 을 확인 할 수 있다. (slave2)
>>> single thread가 오히려 slave2 보다 QPS가 약간 높게 나왔다. (slave3)
0
5000
10000
15000
20000
25000
insertQPS
Case #3 – 1 database,TABLE

23
> Case #4 – 4 databases
>> Performing simple insert query in 16 threads on masterDB (4 databases, TABLE)
>>> database가 4개 일때 worker thread로 parallel 하게 처리되는 것을 확인 할 수 있다. (slave2)
>>> single thread 처리는 database가 1개 일때와 동일한 결과를 가져온다. (slave3)
0
5000
10000
15000
20000
25000
insertQPS
Case #4 – 4 database,TABLE

24
> Case #1_#2 – 1 database + FILE vs 4 databases + FILE : QPS
MASTER
SLAVE 1
(logical_clock)
SLAVE 2
(database)
SLAVE 3
(single)
44m/45m
70m/75m
298m/105m
3times
276m/277m

MASTER
SLAVE 1
(logical_clock)
SLAVE 2
(database)
SLAVE 3
(single)
40%
14%/27%
13%
45%
> Case #1_#2 – 1 database + FILE vs 4 databases + FILE : CPU

MASTER
SLAVE 1
(logical_clock)
SLAVE 2
(database)
SLAVE 3
(single)
45m/45m
76m/76m
370m/124m
3times
342m/340m
> Case #1_#2 – 1 database + TABLE vs 4 databases + TABLE : QPS

LOAD TEST II - binlog_group_commit_sync_delay
Master - - SET GLOBAL binlog_group_commit_sync_delay = 10; FILE/TABLE
STOP SLAVE;
START SLAVE;
FILE/TABLE
STOP SLAVE;
START SLAVE;
FILE/TABLE
sysbench --test='/usr/share/doc/sysbench/tests/db/oltp.lua' --oltp_tables_count=1
--report-interval=10 --oltp-table-size=3000000 --num-threads=16 --mysql-user=root
--mysql-password=test --mysql-table-engine=innodb --rand-init=on
--mysql-socket=/mysql/MyHome/tmp/mysql.sock
--mysql-host=localhost --mysql-port=20306 --mysql-db=sysbench
prepare
mysqldump --skip-extended-insert --no-create-info -p'test' sysbench > dumpfile.sql

28
>> Performing simple insert query in 16 threads (1 database,FILE)
>>> master서버는 binlog_group_commit_sync_delay 설정으로 master에서 동시에 처리할 수 있는 QPS가
줄어 든것을 확인할 수 있다. 특이사항은 Slave1의 경우는 master보다 동시에 처리할 수 있는 TRX가 증가하
였다. Slave2,slave3은 이전 테스트결과와 동일하게 나왔다.
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
insertQPS

29
> Case #5 – 1 database & FILE - QPS
MASTER
SLAVE 1
(logical_clock)
SLAVE 2
(database)
SLAVE 3
(single)
60m (44m)
72m (70m)
291m (298m)
270m (276m)

TO DO 30
• MTS GTID vs binary file based
• MTS Configure Real service
• Rolling out GTID and MTS to a live system with no
downtime
• 데이터 암호화 / 특정 테이블 Load Test
• MTS 사용시 백업 이슈는 없는가?

Reference 31
# MySQL Replication and Multi-threaded Slaves
http://www.slideshare.net/shiv4289/mysql-user-campmultithreadedslaves
# The latest with MySQL on OpenStack Trove
http://www.slideshare.net/tesoracorp/the-latest-with-mysql-on-openstack-trove
# MySQL Replication and Scalability
http://www.slideshare.net/shiv4289/my-sql-replicationscalability
# MySQL 5.6 replication
http://www.slideshare.net/MarkSwarbrick/mysql-56-replication-webinar
# Performance issues and fixes -- MySQL 5.6 Semi-Synchrnous Replication
http://yoshinorimatsunobu.blogspot.jp/2015/01/performance-issues-and-fixes-mysql-56.html
# Q&A: Multi-threaded Replication in MySQL 5.6 and MySQL 5.7
https://www.percona.com/blog/2015/03/13/qa-multi-threaded-replication-in-mysql-5-6-and-mysql-5-7/
> MTS

MySQL 5.7 MTS (Multi Thread Slave)

Recomendados

Recomendados

Más contenido relacionado

Más de I Goo Lee

Más de I Goo Lee (20)

Último

Último (20)

MySQL 5.7 MTS (Multi Thread Slave)