Доклад Алексея Копытова на HighLoad++ 2014.

1. Ïðîáëåìû èñïîëüçîâàíèÿ MySQL íà ñîâðåìåííîì
îáîðóäîâàíèè
Alexey.Kopytov@percona.com
October 26, 2014

2. Êòî ÿ? Ãäå ÿ?
Àëåêñåé Êîïûòîâ <alexey.kopytov@percona.com>
I Percona Server
I Percona XtraBackup (ðóêîâîäèòåëü ïðîåêòà)
I Percona XtraDB Cluster

3. Íåìíîãî èñòîðèè
MySQL
I ðàçðàáîòêà íà÷àëàñü â 1995ã.
I íåêîòîðûé êîä áûë íàïèñàí â 1979ã.
I èçíà÷àëüíî
ðàçðàáîòêà âåëàñü íà Solaris/Sparc
InnoDB
I ðàçðàáîòêà íà÷àëàñü â 1995ã.
I âêëþ÷åíà â MySQL â 2000ã.
I èçíà÷àëüíî
ðàçðàáîòêà âåëàñü íà Windows NT
Còàðûå ïðîåêòû ïî ìåðêàì IT

4. Êðèòè÷åñêèå ó÷àñòêè êîäà ïèñàëè ïîä ñîâñåì äðóãîå
“æåëåçî”:
/*************************************************************//**
Runs an idle loop on CPU. The argument gives the desired
delay in microseconds on 100 MHz Pentium + Visual C++. */
...
/* Semaphore operations in operating systems are slow:
Solaris on a 1993 Sparc takes 3 microseconds (us) for a
lock-unlock pair and Windows NT on a 1995 Pentium takes 20
microseconds for a lock-unlock pair. */

5. Ïðîáëåìû ñ ñîâðåìåííûì ¾æåëåçîì¿:
I ïàìÿòü (áîëüøèå îáú¼ìû, NUMA âñòðå÷àåòñÿ âñ¼ ÷àùå)
I ïðîöåññîðû: ¾áîëüøå ëîøàäèíûõ ñèë¿, íî èñïîëüçóþòñÿ íåýôôåêòèâíî
I SSD ÷àñòî òðåáóþò èíîãî ïîäõîäà äëÿ ÷òåíèÿ/çàïèñè äàííûõ
I áëîêèðîâêè â ñåðâåðå íåàäåêâàòíû âûñîêèì íàãðóçêàì

6. jemalloc
I ìàñøòàáèðóåìàÿ ðåàëèçàöèÿ malloc()
http://www.canonware.com/jemalloc/
I öåëåâàÿ íàãðóçêà: Nthreads > Ncores
I èñïîëüçóåòñÿ ïî óìîë÷àíèþ â Firefox, FreeBSD, NetBSD
I îáÿçàòåëüíîå òðåáîâàíèå äëÿ TokuDB

7. jemalloc
I ñâåæèå âåðñèè â ðåïîçèòîðèÿõ Percona
I îïöèÿ malloc-lib â my.cnf:
[mysqld_save]
malloc-lib=/usr/lib/mysql/libjemalloc.so

8. Buffer pool

9. Buffer pool: êîíòðîëüíûå ñóììû
I ïðîáëåìà ïðè áîëüøèõ IOPS (SSD)

10. Buffer pool: êîíòðîëüíûå ñóììû
Êàê îáíàðóæèòü?
I buf_calc_page_new_checksum â PMP, OProfile, perf
I Performance Schema íèêàê
×òî äåëàòü?
I innodb_fast_checksum=1 â Percona Server 5.1/5.5
I innodb_checksum_algorithm=crc32 â Percona Server è MySQL 5.6

11. Buffer pool: êîíòðîëüíûå ñóììû
innodb_checksum_algorithm=crc32
I âûêëþ÷åíî ïî óìîë÷àíèþ
I íå strict_crc32!
I èñïîëüçóåò àïïàðàòíîå óñêîðåíèå (åñëè äîñòóïíî)
I ðàáîòàåò òîëüêî äëÿ îáíîâë¼ííûõ ñòðàíèö!
I ïî÷åìó-òî îñòàâèëè ìåäëåííûé àëãîðèòì äëÿ redo log

12. Buffer pool: NUMA
Uniform Memory Access vs Non-Uniform Memory Access:

13. Buffer pool: NUMA
I âñåãäà åñòü îáðàùåíèÿ ê ÷óæîé ïàìÿòè

14. Buffer pool: NUMA
I ïàìÿòü âûäåëÿåòñÿ íà òåêóùåé íîäå ïî óìîë÷àíèþ

15. Buffer pool: NUMA
I ÷åðåäîâàíèå (interleaving) ïàìÿòè:
# numactl --interleave all mysqld ...
I interleaving íà óðîâíå BIOS – ïëîõî!

16. Buffer pool: NUMA
I ôàéëîâûé êýø òîæå ìåøàåò:
I ðåøåíèå:
# sysctl -q -w vm.drop_caches=3

17. Buffer pool: NUMA
Linux memory overcommit:
I ïàìÿòü âûäåëÿåòñÿ òîëüêî â ìîìåíò äîñòóïà
I ïðè ñòàðòå mysqld íóæíî ñäåëàòü memset() ïîêà ôàéëîâûé êýø ïóñòîé

18. Buffer pool: NUMA
Êàê îáíàðóæèòü?
I ¾À-à-à, òîðìîçèò âñ¼!¿
I áîëüøîé ðàçáðîñ â ðåçóëüòàòàõ
I swapping ïðè ñâîáîäíîé ïàìÿòè

19. Buffer pool: NUMA
Ïàò÷è îò Twitter â Percona Server 5.5 / 5.6:
I îïöèè mysqld_safe:
[mysqld_save]
numa_interleave=1
flush_caches=1
I îïöèè ñåðâåðà:
[server]
innodb_buffer_pool_populate=1

20. Buffer pool: ïðîáëåìû ñ DROP/TRUNCATE TABLE
innodb_file_per_table=1 + áîëüøîé buffer pool
óäàëåíèå ñòàðûõ ñòðàíèö èç buffer pool:

21. Buffer pool: ïðîáëåìû ñ DROP/TRUNCATE TABLE
bug #51325 “Dropping an empty innodb table takes a long time with
large buffer pool”
I innodb_lazy_drop_table=1 â Percona Server 5.1/5.5
I ïîçæå èñïðàâëåí â MySQL 5.1/5.5/5.6
I O(n), íî buffer pool ìüþòåêñ ïåðèîäè÷åñêè îòïóñêàþò
bug #61188 “DROP TABLE extremely slow”
I òà æå ïðîáëåìà, íî äëÿ êîìïðåññèðîâàííûõ òàáëèö
I èñïðàâëåíî â MySQL 5.1/5.5/5.6
TRUNCATE TABLE
I áóäåò èñïðàâëåíî òîëüêî â MySQL 5.7

22. Buffer pool: ãëîáàëüíûé ìüþòåêñ

23. Buffer pool: ãëîáàëüíûé ìüþòåêñ
Ðåøåíèå â MySQL 5.5:
innodb_buffer_pool_instances= N

24. Buffer pool: ãëîáàëüíûé ìüþòåêñ
Ðåøåíèå â MySQL:
ïðîáëåìû:
I íåðàâíîìåðíîå èñïîëüçîâàíèå (íåò áàëàíñèðîâêè)
I ðåøàåò ïðîáëåìó ëèøü ÷àñòè÷íî
I èíîãäà äàæå óõóäøàåò ïðîèçâîäèòåëüíîñòü (Bug #67808)

25. Buffer pool: ãëîáàëüíûé ìüþòåêñ
Ðåøåíèå â Percona Server
I êîìáèíèðóåòñÿ ñ innodb_buffer_pool_instances

26. Buffer pool: ãëîáàëüíûé ìüþòåêñ
Ðåøåíèå â Percona Server
I ðåàëèçîâàíî åù¼ äëÿ 5.1
I âñ¼ åù¼ àêòóàëüíî äëÿ 5.6

27. MVCC: kernel_mutex è trx_sys-mutex
read views:

28. MVCC: kernel_mutex è trx_sys-mutex
read views:

29. MVCC: kernel_mutex è trx_sys-mutex
ñîçäàíèå read view:

30. MVCC: kernel_mutex è trx_sys-mutex
read-only òðàíçàêöèè â MySQL 5.6:
I SELECT + AUTO_COMMIT=1 èëè START TRANSACTION READ ONLY
I read-only òðàíçàêöèè íå äîáàâëÿþòñÿ â trx_list
I íåò ïðîáëåìû, åñëè trx_list ïóñòîé (ò.å. åñëè íåò äðóãèõ òðàíçàêöèé!)
áóäåò â MySQL 5.7:
I + àâòîîïðåäåëåíèå äëÿ íå AUTO_COMMIT òðàíçàêöèé

31. MVCC: trx descriptors
I â Percona Server 5.5 / 5.6:
I â MySQL 5.7 áóäåò òî æå ñàìîå

32. MVCC: trx descriptors

33. MVCC: kernel_mutex è trx_sys-mutex
Êàê îáíàðóæèòü?
I SHOW ENGINE INNODB STATUS:
--Thread 140370743510784 has waited at trx0trx.c line 1184 for 0.0000 seconds the semaphore:
Mutex at 0x2b0ccc8 ’kernel_mutex’, lock var 1
...
I PMP:
234 ... innobase_commit_low,innobase_commit...
I PERFORMANCE_SCHEMA:
I wait/synch/mutex/innodb/kernel_mutex (5.5)
I wait/synch/mutex/innodb/trx_sys_mutex (5.6)
×òî äåëàòü?
I ïîïðîáîâàòü jemalloc
I èñïîëüçîâàòü read-only òðàíçàêöèè, åñëè åñòü âîçìîæíîñòü
I ïåðåéòè íà Percona Server èëè MariaDB + XtraDB

34. Adaptive Hash Index:
óçêîå ìåñòî íà AHI mutex:
êàê îáíàðóæèòü:
I PMP, OProfile, perf: btr_search_..., row_search_for_mysql
I PERFORMANCE_SCHEMA: wait/synch/rwlock/innodb/btr_search_latch

35. Adaptive Hash Index
Ðåøåíèå â Percona Server:
innodb_adaptive_hash_index_partitions = N
I ôðàãìåíòàöèÿ ðåñóðñîâ
I ðàáîòàåò òîëüêî ïðè äîñòóïå ê íåñêîëüêèì òàáëèöàì

36. index lock
2 ìåõàíèçìà îáíîâëåíèÿ B-Tree èíäåêñîâ:
I îïòèìèñòè÷åñêèé ñòðàíèöà îáíîâëÿåòñÿ ïî ìåñòó
I ïåññèìèñòè÷åñêèé ðåîðãàíèçàöèÿ ñòðàíèö = áëîêèðîâêà âñåé òàáëèöû!
I ìîæåò ÷èòàòü ñòðàíèöû ñ äèñêà, ïîêà èíäåêñ çàáëîêèðîâàí
Êàê îïðåäåëèòü?
I PMP: btr_cur_search_to_nth_level()
I PERFORMANCE_SCHEMA: new_index-lock
×òî äåëàòü?
I èñïîëüçîâàòü partitioning
I îáåùàþò èñïðàâèòü ¾â áîëüøèíñòâå ñëó÷àåâ¿ â MySQL 5.7

37. Redo log: ALL_O_DIRECT
Ïðîáëåìà:
Ðåøåíèå â Percona Server:
innodb_flush_method=ALL_O_DIRECT

38. Redo log: ïðîáëåìà ñ ÷òåíèåì ïðè çàïèñè
I Ðàçìåð redo log áëîêà 512 áàéòîâ
I Ðàçìåð ñòðàíèöû ïàìÿòè – 4 êèëîáàéòà
I åñëè ñòðàíèöà íå â êýøå, íóæíî ñíà÷àëà ïðî÷èòàòü îñòàâøóþñü ÷àñòü

39. Redo log: ïðîáëåìà ñ ÷òåíèåì ïðè çàïèñè
Êàê ðåøàòü?
Ðåøåíèå â Percona Server 5.1/5.5/5.6:
I óâåëè÷èâàåì log block äî ðàçìåðà ñòðàíèöû
innodb_log_block_size=4096
Ðåøåíèå â MySQL 5.7:
I çàïèñûâàåì áëîê â 512 áàéòîâ è 3.5K íóëåé:
innodb_log_write_ahead_size=4096
Áîëüøå IOPS íà SSD!

40. Redo log: êîíòðîëüíûå ñóììû
Êëèåíò: õîòèì çàãðóæàòü 130mil çàïèñåé î÷åíü áûñòðî
I LOAD DATA INFILE
I íåñêîëüêî ïîòîêîâ
I áåç secondary indexes
I ìíîãî partitions
I ñåé÷àñ 350k rows/second
I íóæíî â 2 ðàçà áûñòðåå

41. Redo log: êîíòðîëüíûå ñóììû
Çàòîð íà log_sys-mutex:
ulint
log_block_calc_checksum(
const byte* block) /*! in: log block */
{
for (i = 0; i OS_FILE_LOG_BLOCK_SIZE - LOG_BLOCK_TRL_SIZE; i++) {
ulint b = (ulint) block[i];
sum = 0x7FFFFFFFUL;
sum += b;
sum += b sh;
sh++;
if (sh 24) {
sh = 0;
}
}
return(sum);
}

42. Redo log: êîíòðîëüíûå ñóììû
ïî÷åìó áû íå èñïîëüçîâàòü àïïàðàòíûå êîíòðîëüíûå ñóììû (êàê
äëÿ ñòðàíèö â buffer pool)?
I 350k rows/sec = 800k rows/sec!
â Percona Server ïîÿâèëàñü îïöèÿ
innodb_log_checksum_algorithm = {none, crc32, innodb}

43. Redo log: êîíòðîëüíûå ñóììû
Êàê îïðåäåëèòü?
I PERFORMANCE_SCHEMA: log_sys_mutex
I PMP:
33 log_block_calc_checksum,log_block_store_checksum,...
Êàê ðåøàòü?
I innodb_log_checksum_algorithm=crc32 â Percona Server 5.6

44. SSD: ðàçìåð ñòðàíèöû
ñòàíäàðòíûé ðàçìåð InnoDB ñòðàíèöû (16KB)
Percona Server 5.1, 5.5, 5.6, MySQL 5.6: innodb_page_size=4096
+:
I ìåíüøå read/write amplification
I +10% íà íåêîòîðûõ I/O bound íàãðóçêàõ
:
I áîëüøèé îáú¼ì äàííûõ íà äèñêå ( 20%)
I ìàêñèìàëüíûé ðàçìåð çàïèñè 2KB
I èçáûòî÷íàÿ çàïèñü â doublewrite: bug #69842

45. SSD:
Íå èìåþò ñìûñëà äëÿ SSD
I ðàçìåð ñòðàíèöû â 16KB;
I ðàçìåð redo log áëîêà â 512B
I read-ahead

46. Îäíîïîòî÷íàÿ ïðîèçâîäèòåëüíîñòü
Ìàñøòàáèðóåìîñòü íà 100+ ñîåäèíåíèé âàæíà, íî:
I ïðîèçâîäèòåëüíîñòü â 1 ñîåäèíåíèå == âðåìÿ îòêëèêà
I ðåïëèêàöèÿ ðàáîòàåò â îäèí ïîòîê (ïî÷òè)
I àäìèíèñòðàòèâíûå çàäà÷è – êàê ïðàâèëî îäèí ïîòîê
I â áîëüøèíñòâå ñëó÷àåâ ñåðâåð ðàáîòàåò ñ íåáîëüøèì êîëè÷åñòâîì
ñîåäèíåíèé
×òî åñòü ïîêà:
I innodb_page_size=4096
I innodb_log_block_size=4096 (òîëüêî â XtraDB)
I innodb_use_atomic_writes (òîëüêî â XtraDB)
I àäåêâàòíûé flushing (XtraDB, MySQL 5.7)
ðàáîòàåì ñ ïðîèçâîäèòåëÿìè SSD

47. Îäíîïîòî÷íàÿ ïðîèçâîäèòåëüíîñòü
I íàãðóçêè ñ êîëè÷åñòâîì àêòèâíûõ ñîåäèíåíèé 100
I ïðàêòè÷åñêè íèêàêèõ óñèëèé ïî îïòèìèçàöèè
I êàæäûé ñëåäóþùèé ðåëèç õóæå ïðåäûäóùåãî
I ìíîãî ïîäðîáíîñòåé â áëîãå Mark Callaghan:
http://smalldatum.blogspot.com

48. Îäíîïîòî÷íàÿ ïðîèçâîäèòåëüíîñòü
Èñòî÷íèê: http://smalldatum.blogspot.com

49. Âîïðîñû!