Dd and atomic ddl pl17 dublin

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0:
Atomic DDLs – Implementation and Impact
Ståle Deraas, Senior Development Manager
Oracle, MySQL
26 Sept 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

Agenda
Requirements for a Data Dictionary Storage Engine
Requirements for an SE implementing Atomic DDLs
Effects of new INFORMATION_SCHEMA in MySQL 8.0
1
2
3

Requirements for a Data Dictionary Storage
Engine

MySQL Data Dictionary before MySQL 8.0
5
Data Dictionary
Files
FRM TRG OPT
System Tables (mysql.)
user procevents
InnoDB System Tables
MyISAM
File system
InnoDB
SQL

Transactional Data Dictionary in MySQL 8.0
6
Data Dictionary
DD SE
SQL
DD TableDD TableDD Table

Requirements for Data Dictionary Storage Engine
• Some special care need to be taken at initialize/bootstrap
• Must support transactions
• Must support “attachable transactions”
– To open a table, the DD SE needs to open DD tables
• In MySQL 8.0 InnoDB has dual role
– MySQL Data Dictionary Storage Engine (DD SE)
– Storage Engine that supports Atomic DDL
7

The Data Dictionary Storage Engine API
• An addition to the Storage Engine API
• Not strictly separated from SE API
• Some examples of functionality that the DD SE must implement:
- Initialize dictionary support
- Initiate recovery of dictionary transactions
- Reset of the DD SE internal cache of meta data
8

MySQL Server
Data Dictionary Storage Engine
9
Query Executor
Optimizer
Storage
Engine
SQL
statement
Client
Parser
Result Data
Dictionary
DD Storage
Engine

Example - DROP SCHEMA at high level
10
MySQL 5.7
• Delete tables
– Metadata, TRN/TRG/FRM files
– Data, InnoDB tables
• Delete stored programs
– Metadata, rows in MyISAM (non-
transactional)
• Delete schema
– Metadata, DB.OPT file
Mix of filesystem, non-
transactional/transactional
storage and multiple commits
MySQL 8.0
• Delete tables
– Metadata, rows in InnoDB
– Data, InnoDB tables
• Delete stored programs
• Delete schema
Updates to transactional storage,
one commit

Requirements for an SE implementing
Atomic DDLs

Why Atomic DDL support in MySQL 8.0
• Ensure we have a consistent state after a DDL operation
– Prevent slave drift
– Prevent internal inconsistencies
• At last possible due to transactional storage of meta data
• Enables implemention of crash-safe DDL
• Enables addressing longstanding issues
12

User visible changes to Atomic TABLE DDL in MySQL 8.0
• DROP TABLES , all tables dropped or none
• DROP SCHEMA, all entities in the schema are dropped, or none
• Note that atomic DDL statements will be rolled back or committed even in
case of crash, e.g. RENAME TABLES
• CREATE TABLE would be successfully committed or rolled back (no orphan
ibd left)
• TRUNCATE TABLE (including InnoDB tables with FTS AUX tables) would be
successfully committed or rolled back
• RENAME TABLES, all or none
• ALTER TABLE successful or not done
13

Atomic DDL: Storing SE private data (prerequisite)
• An SE which will support atomic DDL can store private data in SE private
columns of Data Dictionary tables
• InnoDB has chosen to store key-value pairs in the SE private data
• Examples: index root page number, InnoDB table ID, tablespace ID etc.
14

How do we implement Atomic and Crash-Safe TABLE DDL
• At high level, create a single atomic transaction:
– Do updates to the Data Dictionary (DD SE)
– Do necessary changes in the Storage Engine
– Do necessary writes to binary log
– Since it is a single transaction, all updates to Data Dictionary can be rolled back and
are crash safe
15

How do we implement Atomic and Crash-Safe TABLE DDL
• At more detailed level:
– Make sure there are no intermediate commits on SQL layer during DDL
– SE methods should have no intermediate commits, and register as part of trx
– SE can do redo/rollback of DDL , implemented with DDL_LOG in InnoDB
– SQL layer will invoke a post_ddl() hook, so SE can do post commit/rollback work
– As part of the transaction, write to binary log
– Introduced the SE property HTON_SUPPORTS_ATOMIC_DDL, that can be tested for in
the source code
16

InnoDB: Features in SE for Atomic and Crash-Safe TABLE DDL
• DDL_LOG
– Protected table in mysql tablespace, so no DDL and no user DML allowed
– Add trx Ids to all entries, which will be deleted by post_ddl() hook
– Changes to DDL_LOG are persisted ASAP. Will prevent data files updates and no
flushing of DDL_LOG entries (exempted from innodb_flush_log_at_trx_commit
control for delayed flush)
• Issue with physical files and index tree:
– Before MySQL 8.0, could leave “orphan” InnoDB ibd files, “orphan” index trees/pages
– Orphan files/trees cause problems for subsequent DDLs and waste tablespace
– With MySQL 8.0, DDLs “crash safe”, no ibd files/index trees orphaned if no commit
– At commit, the old ibd files/index trees will be dropped
17

Atomic DDL: Use Case - CREATE TABLE, SQL Perspective
18
MySQL 5.7
• At SQL layer: create table obj
• Store obj in DD tables, commit
• «Open» table, construct all
internal structures
• Call to SE
handler::create(name,...)
MySQL 8.0
• At SQL layer: create table obj
• Call to SE, adding SE private data
• Store obj in DD tables *
• «Open» table, construct all internal
structures
• Call to SE handler::create(name,...),
which can update SE private data
• Store obj in DD tables *
* = commit for non-atomic SE

Atomic DDL: Use Case - CREATE TABLE, SQL Perspective CONT
19
MySQL 5.7
• SE creates physical files, its own
artifacts such as InnoDB
tablespaces. InnoDB stores data
in own system tables.
• Write statement to binary log
MySQL 8.0
• Write statement to binary log
• Commit/Rollback, post_ddl() hook

Atomic DDL: Use Case - CREATE TABLE, SE perspective
20
MySQL 5.7
• Call to SE handler::create(name,.)
• SE creates the physical
tablespaces/files(file-per-table) or
Cluster index tree and other index
trees
• SE starts its own transaction,
insert the new table/indexes's
metadata to InnoDB own System
tables
MySQL 8.0
• Call to SE handler::create(name,...)
• SE creates the physical
tablespaces/files(file-per-table) or
Cluster index tree and other index
trees
• SE logs above physical file and indexes
creation
• Note: No separate SE transaction, all
info for new tablespace/indexes are
passed back to server with DD objects

Atomic DDL: Use Case - CREATE TABLE, SE perspective cont
21
MySQL 5.7
• Note: If crash happens before
above trx commit, there could be
orphan files or index trees left
MySQL 8.0
• SQL layer commits or rollback
• Call to SE post_ddl(). If rollback, the
post_ddl() physically delete the
tablespace/ibd (file-per-table) and
drop the index trees for the table
• Note: If crash, SE will recover info in
DDL_LOG to delete the ibd/index trees

Atomic DDL: Use Case - DROP TABLES, SQL Perspective
22
MySQL 5.7
• For each table in table list
– Call SE handler:ha_delete_table()
and if error do appropriate action
– Remove for Data Dictionary and
commit
• Write up to 3 artifical DROP
TABLES to binary log for
dropped tables (temp tables *2
+ all base tables)
MySQL 8.0
– For each table in table list
• Check for 5 types of tables
– Err out if non-existing tables and no IF
EXISTS clause
– Handle tables in non-atomic DDL SE
• Delete Table in SE, and del in DD, commit
– Handle tables in atomic DDL SE
• Delete Table in SE, and del in DD, no commit
– Write DROP TABLES statment to binary log
incl all table (in «no GTID» mode)
– Commit/rollback , post_ddl() hook
• Concurrent DDL on table blocked

Atomic DDL: Use Case - DROP TABLES, SQL Perspective cont
23
MySQL 5.7 MySQL 8.0
– If GTID mode, and write to binlog
postponed, write DROP TABLES stmt
with all tables we dropped to binlog
• To handle GTID mode from older servers
– For each temp table in non-
transactional SE
• Drop table, call SE handler::delete_table()
– Write DROP TEMPORARY TABLES for all
temp tables above to binary log
– For each temp table in transactional SE
• Drop table, call SE handler::delete_table()
– Write DROP TEMPORARY TABLES for all
temp tables above to binary log

Atomic DDL: Use Case - DROP TABLES, SE perspective
24
MySQL 5.7
• Call to SE
handler:ha_delete_table()
• InnoDB starts its own transaction
and delete metadata from the
InnoDB own system tables
• InnoDB commit trx
• InnoDB deletes physical files/ibd
(in file-per-table case)
• Note: If crash happens after
InnoDB commit trx, there could
be orphaned files/ibd or trees
MySQL 8.0
• Call to SE handler:ha_delete_table()
• InnoDB only logs actions to delete ibd
file or index trees in DDL_LOG, so no
physical action (No InnoDB system
tables needs to be updated)
• Back to SQL layer to commit - Delete
metadata in DD system tables and
commit

Atomic DDL: Use Case - DROP TABLES, SE perspective cont
25
MySQL 5.7 MySQL 8.0
• SQL layer calls post_ddl(). If commit,
InnoDB physically deletes ibd and drop
index trees. If rollback, delete entries
in DDL_LOG, files and trees left as they
are
• Note: If crash and DROP trx
committed, recover info from
DDL_log, and delete files/trees during
crash recovery

How do we implement Atomic and Crash-Safe non-TABLE DDL
• At slightly detailed level (this is all at SQL layer):
– Make sure there are no intermediate commits during DDL
– Write to binary log as part of the DDL transaction
– Ensure caches for Data Dictionary, routines, events and UDFs are consistent with DDL
status
– Ensure user visible behaviour is atomic
– NOTE: Behaviour of DROP VIEW changes, for list of views
• Pre-MySQL 8.0: Giving an error for non-existent views and removing existing views
• MySQL 8.0: Giving an error for non-existent views
26

Atomic DDL Non-TABLE: Use case - CREATE ROUTINE
27
MySQL 5.7
• Create routine obj
• Store obj in DD tables, commit
• For SF, find VIEWs usage and
update view cols metadata for
each view. Commit for each
view.
• Invalidate cache
• Write stored routine statement
to binary log
MySQL 8.0
• Create routine obj
• Store obj in DD tables
• For SF, find VIEWs usage and
update view cols metadata for each
view
• Write stored routine event to
binary log
• Commit and invalidate cache, or on
error, rollback and give msg

Atomic DDL for user management: Overview
28
MySQL 5.7
• Partial execution is allowed
– Make persistent changes to ACL
tables/caches
– Throw error
• Error propagation to binary log
– Required for HA setup
– Other nodes expect same error
while executing the statement
MySQL 8.0
• Execution is atomic
• A statement either fully succeeds or
is rolled back
• No error propagation to binary log
• Only a successful statment makes it
way to binary log
• Simpler handling in HA setup

Atomic DDL for user management: Use Case - CREATE USER
29
MySQL 5.7
• Result
– foo@mysql.com is created
– Error is thrown for
bar@mysql.com for invalid
authentication plugin
• Binary log
– Statement is written to binary log
with password transformation
– Expected error for
bar@localhost.com is written
MySQL 8.0
• Result
• Error because of invalid
authentication plugin for
bar@localhost.com
• foo@mysql.com is not created
• Binary log
• No statement is written to binary log
CREATE USER `foo`@`mysql.com` IDENTIFIED BY ‘haha’,
`bar`@`mysql.com` IDENTIFIED WITH ‘non_existing_plugin’ BY ‘hoho’;

Atomic DDL for user management: Use Case - GRANT
30
MySQL 5.7
• Result
– foo@mysql.com get ability to
SELECT from all tables/views.
– Error is thrown for non existing
user
• Binary log
– Statement is written to binary log
– Expected error for
`non_existing`@`mysql.com`
MySQL 8.0
• Result
• Error because of invalid user
• foo@mysql.com is not granted
SELECT
• Binary log
• No statement is written to binary log
GRANT SELECT ON *.* TO `foo`@`mysql.com`, `non_existing@mysql.com`;

Atomic DDL for user management: Note that...
31
• IF EXISTS extensions (ALTER USER | DROP USER)
– Will continue to ignore errors related to non-existing authorization IDs
– All other errors will cause statement rollback
• IF NOT EXISTS extensions ( CREATE USER | CREATE ROLE)
– Will continue to ignore errors related to existing authorization IDs
– All other errors will cause statement rollback
• New MDL type to protect access to in-memory caches for user
management

Effects of new INFORMATION_SCHEMA in
MySQL 8.0

NEW INFORMATION_SCHEMA in MySQL 8.0
33
Uniform, simpler implemention makes it a lot faster
MySQL Client
I_S Query Results
MySQL Server
Optimizer prepares
execution plan.
Executor reads metadata from
data dictionary tables.
InnoDB storage engine
Return rows to user.
INFORMATION_SCHEMA in 8.0
MySQL Client
I_S Query Results
MySQL Server
Create temporary table.
Heuristic optimization.
Read metadata from File system or
from MyISAM/InnoDB engine.
.
TEMP TABLE
Return rows to user.
INFORMATION_SCHEMA in 5.7
File system / MyISAM
/ InnoDB engine

INFORMATION_SCHEMA Performance and Scalability
• Typically 30X performance improvements over MySQL 5.7
• More than 100X for some queries like: List all InnoDB table columns
34
I_S queries scale, both with database size and query load
0 20 40 60 80 100 120 140 160
List all InnoDB tables columns 5k tables
List all InnoDB tables columns 10k tables
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 35
100 schemas times 50 tables (5000 tables)
INFORMATION_SCHEMA Performance
0 0.5 1 1.5 2 2.5 3 3.5 4
Count All Schemas
Schema aggregate size stats
All Dynamic Table Info
All Static Table Info
Auto Increments Near Limit
Count All Columns
Count All Indexes
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)

INFORMATION_SCHEMA in MySQL 8.0, community feedback
• In MySQL 8.0 DMRs : Option “information_schema_stats” introduced
– TABLES.TABLE_ROWS, TABLES.DATA_FREE, …
• Handled by opening table and retrieving data from SE in MySQL 5.7 – expensive!!!
• Bugreport: Nicolai Plum bug#83957 and Peter Brawley bug#87548
• Feedback from several MySQL ACEs, that found inital solution confusing
• For MySQL 8.0.3 RC1 We have created an improved solution
36

INFORMATION_SCHEMA in MySQL 8.0.3
• New option: information_schema_stats_expiry
– Default 24h
– Cache dynamic values in mysql.index_stats and mysql.table_stats
– Set to 0, always fetch data from SE and no «stats» will be stored
– ANALYZE will update mysql.index_stats and mysql.table_stats
• Improved performance fetching dynamic data from SE (InnoDB)
– In 8.0.1 DMR, typically 2-3 times faster than 5.7
– In 8.0.3 RC1, typically 4-5 times faster than 5.7
• Option “information_schema_stats” removed
37

Try for yourself!
• Downloadable 8.0.3 RC1
– dev.mysql.com
• Enjoy and give us your feedback!
• Thank you for listening
• http://mysqlserverteam.com
• For details on atomic DDL:
– WL#7743, WL#9173, WL#9045, WL#9175, WL#9045, WL#9536
38

Dd and atomic ddl pl17 dublin

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Dd and atomic ddl pl17 dublin

Similar a Dd and atomic ddl pl17 dublin (20)

Último

Último (20)

Dd and atomic ddl pl17 dublin