Making Backups in Extreme Situations

Safe Harbour Statement
THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT
DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY,
AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A
COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR
FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING
PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND
TIMING OF ANY FEATURES O FUNCTIONALITY DESCRIBED FOR
ORACLE’S PRODUCTS REMAINS AT THE SOLE DISCRETION OF
ORACLE.


Making Backups in Extreme
Situations
Sveta Smirnova
Principal Technical Support Engineer
Matt Lord
MySQL Product Manager


Insert Picture Here

Program Agenda









Introduction
Agenda Overview
Dealing With Space Constraints
Dealing With Time Constraints
Dealing with Resource Usage Constraints
Minimizing Locking Impacts
Handling Data Corruption
Conclusion


Introduction
Insert Picture Here


Who We Are
Matt Lord
 Former MySQL Support engineer
 Worked with hundreds of backup issues
 Now Product Manager

Sveta Smirnova
 MySQL Support engineer
 Support representative in MySQL Enterprise Backup (MEB) team


Why Backups?
1. They are a critical insurance against:
 Hardware failure
 User error
 Program error

2. You can use them to deploy new instances
3. Setup replication clones
4. Data is your most valuable IP. You have to protect it from loss.


The Backup Conundrum
1. Backup tools guarantee consistency by:
 Locking tables
 Running long transactions
 Even using global locks

2. Backup jobs generate a lot of system overhead
3. Backups take up a lot of storage space
How do you backup your critical production systems then?





Servers are performing critical business functions
Downtime is not acceptable
Negative performance impacts are not acceptable
Storage space is critical


So How Do I Perform Backups...
 without using local storage?
 with minimal local resource usage?
 with minimal impact on the MySQL instance I'm backing up?
 when I need the backup to complete in a small time window?
 without locking tables and entire MySQL instances?
 when the data or the backup itself has corrupt data?


What We Won't Cover


Full backups



Partial backups



Incremental backups



Backup automation



General backup tuning

Instead, see:


MySQL Enterprise Backup BOF [BOF3558, Saturday, 5:30 PM]



MySQL Backup-and-Recovery [CON7659, Sunday, 11:30 AM]



Backing Up MySQL Databases [CON3278, Sunday, 1:00 PM]



MySQL Enterprise Edition – Hands-On Lab, Sunday, 10:00 AM


Dealing With Storage
Constraints


Insert Picture Here

When There Is Little Storage Space
 Very large data directory size
 Small amount of local storage space left
 But MEB process should be run on the same machine!


Backup Options
 Compressed backups: --compress, --compress-level=LEVEL
 Stream backups over SSH
 Backup directly to tape, using the SBT interface
 Create partial backups: --include, --with-tts
 Create incremental backups
 Don’t backup empty and unused pages: --skip-unused-pages


Backup Options: Compression


Compression ratios of 80% or more are possible!

shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed
shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed
> backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed
> backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed
334M
/opt/mysql/backups/uncompressed
334M
/opt/mysql/backups/uncompressed
334M
total
334M
total
shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed
>> --compress --compress-level=9 backup 2>/dev/null && du –csh
--compress --compress-level=9 backup 2>/dev/null && du –csh
>> /opt/mysql/backups/compressed
/opt/mysql/backups/compressed
81M
81M
81M
total
81M
total


Backup Options: Stream Backups


We can avoid local storage altogether

Simply specify the tmpdir as the backup location (for temporary data)
 Should be big enough to keep all changes made during backup
 Still no space requirements for data files
shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup
shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup
> --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo
> --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo
> "cat > /tmp/backup.img“
> "cat > /tmp/backup.img“
shell# ssh matt@solo "ls -lh /tmp/backup.img"
shell# ssh matt@solo "ls -lh /tmp/backup.img"
-rw-r--r-staff
333M Sep 10 15:54 /tmp/backup.img
-rw-r--r-- 11 matt
matt
staff
333M Sep 10 15:54 /tmp/backup.img


Backup Options: Using the SBT Interface
 We can avoid local storage altogether
 Backup directly to your company’s MMS
– Oracle Secure Backup (fully supported)
– IBM Tivoli Storage Manager
– Symantec NetBackup
shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08
shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08
>> --backup-dir=/tmp/backup backup-to-image
--backup-dir=/tmp/backup backup-to-image
shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup
shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup
> --sbt-lib-path=/usr/lib/libobk.so
> --sbt-lib-path=/usr/lib/libobk.so
> --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt”
> --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt”
> --backup-dir=/tmp/backup backup-to-image
> --backup-dir=/tmp/backup backup-to-image


Backup Options: Partial Backups


Only backup your important schemas and tables



MEB now supports InnoDB transportable tablespaces

shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related
shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related
>> --backup-dir=/opt/mysql/backups backup
--backup-dir=/opt/mysql/backups backup
shell# mysqlbackup --use-tts=with-minimum-locking
shell# mysqlbackup --use-tts=with-minimum-locking
> --include=employees.* --backup-dir=/opt/mysql/backups backup
> --include=employees.* --backup-dir=/opt/mysql/backups backup


Backup Options: Incremental Backups


Only backup what’s changed since your last full backup

shell# mysqlbackup --incremental
>> --incremental-base=dir:/opt/mysql/backup/monday
--incremental-base=dir:/opt/mysql/backup/monday
>> --incremental-backup-dir=/opt/mysql/backup/tuesday backup
--incremental-backup-dir=/opt/mysql/backup/tuesday backup
> --incremental-base=history:last_backup --with-timestamp
> --incremental-base=history:last_backup --with-timestamp
> --incremental-backup-dir=/opt/mysql/backup backup
> --incremental-backup-dir=/opt/mysql/backup backup


Backup Options: Skip Blank & Unused Pages
 Tell InnoDB not to backup blank or unused pages
 Use this in combination with compression for optimal storage efficiency

>> --compress --compress-level=9 --skip-unused-pages backup
--compress --compress-level=9 --skip-unused-pages backup


Restore Options
 MEB 3.9 introduces single-step restores
– Even for compressed and single image file backups
 This allows us to avoid unnecessary duplication of data
shell# ssh matt@solo "cat /opt/mysql/backups/backup.img" || mysqlbackup
shell# ssh matt@solo "cat /opt/mysql/backups/backup.img"
mysqlbackup
>> --backup-dir=/tmp/backup --uncompress --backup-image=-
--backup-dir=/tmp/backup --uncompress --backup-image=> --datadir=/var/lib/mysql --innodb_log_group_home_dir=.
> --datadir=/var/lib/mysql --innodb_log_group_home_dir=.
> --innodb_log_files_in_group=4 --innodb_log_file_size=2G
> --innodb_log_files_in_group=4 --innodb_log_file_size=2G
> --innodb_data_file_path="ibdata1:1G:autoextend"
> --innodb_data_file_path="ibdata1:1G:autoextend"
> copy-back-and-apply-log
> copy-back-and-apply-log


Dealing With Time Constraints
Insert Picture Here


I Need the Backup Now!
 Scheduled maintenance windows may be 30 mins or less
 The backup may be needed immediately for a new deployment
 One or more schemas may need to be moved to another host ASAP


Backups Methods
 Direct FS backup
– Requires mysqld to be shutdown, or at least globally locked
 Logical backups (mysqldump)
– VERY slow, consumes a lot of disk space, requires too many locks
 MySQL Enterprise Backup (MEB)
– Provides online hot backups for transactional storage engine InnoDB
– Includes many performance and resource usage controls
– Supports partial backups and transportable tablespaces
– Supports incremental backups


MySQL Enterprise Backup

49x faster than mysqldump

MySQL Enterprise Backup

80x faster than mysqldump

MEB Performance Options
 Increase buffers for CPU intensive multi-threaded ops: --number-of-buffers
 Increase read I/O threads: --read-threads
 Increase write I/O threads: --write-threads
 Increase CPU processing threads: --process-threads
 Increase the maximum memory used for caching: --limit-memory
 Don't backup unused pages in InnoDB tables: --skip-unused-pages


MEB Example: Time Comparison
 2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM
shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp
shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp
backup 2>/tmp/log.txt
backup 2>/tmp/log.txt
real
0m34.403s
real
0m34.403s
user
0m3.069s
user
0m3.069s
sys
0m2.043s
sys
0m2.043s
shell# time mysqlbackup --backup-dir=/opt/mysql/backups
shell# time mysqlbackup --backup-dir=/opt/mysql/backups
> --process-threads=4 --limit-memory=1000 --skip-unused-pages
> --process-threads=4 --limit-memory=1000 --skip-unused-pages
> --with-timestamp backup 2>/tmp/log.txt
> --with-timestamp backup 2>/tmp/log.txt
real
0m29.883s
real
0m29.883s
user
0m3.121s
user
0m3.121s
sys
0m1.889s
sys
0m1.889s

 Cut backup times by 50%+ on enterprise class machines with large datasets

Dealing With Resource Usage
Constraints


Insert Picture Here

Why is the Application Slow Right Now?
 Backups can be CPU intensive
 Backups can be I/O intensive, also causing CPU cycles in IO_Wait
 Backups may use various locks within MySQL
 So how to take backups without having a noticeable affect…


Backup Options
 Limit memory usage: --limit-memory
 Limit CPU usage: --process-threads
 Sleep X ms after each time we copy 1 block of data: --sleep
 Avoid internal locks entirely: --no-locking, --innodb-only
 Limit disk I/O: --read-threads, --write-threads
 Don’t copy empty and unused pages: --skip-unused-pages
 Process limits at OS level: cgroups, taskset, cpulimit, etc.


MEB Example: Resource Usage
 2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM
shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O'
shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O'
>> mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp
mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp
>> backup 2>&1 || tail -1
backup 2>&1
tail -1
Exec Time: 0:23.11, CPU: 21%, MEM: 951648KB, IO: 4229360
shell# taskset -c 1 /usr/bin/time -f
shell# taskset -c 1 /usr/bin/time -f
> 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup
> 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup
> --backup-dir=/opt/mysql/backups --limit-memory=50
> --backup-dir=/opt/mysql/backups --limit-memory=50
> --process-threads=1 --skip-unused-pages --no-locking --sleep=2000
> --process-threads=1 --skip-unused-pages --no-locking --sleep=2000
>> --with-timestamp backup 2>&1 || tail -1
--with-timestamp backup 2>&1
tail -1


Minimizing Locking Impacts
Insert Picture Here


MEB Locking Related Options
 Skip ALL locking: --no-locking (not 100% safe for non-transactional data)
 Only backup transactional data: --innodb-only
 Incremental backups using the redo log: --incremental-with-redo-log-only
 Partial backups with minimal locking: --use-tts=with-minimum-locking


I Thought MEB Backups Were “Hot”?
 FLUSH TABLES WITH READ LOCK is used at the end
– Necessary to get a snapshot of all non-transactional data
 Can be avoided with various options:
– Don’t take any locks at all: --no-locking
– Only backup InnoDB tables: --only-innodb[-with-frm]


What About the Non-Transactional Data?
 mysqlhotcopy – lock and copy each non-transactional table
 mysqldump – lock and logically export each table
 FS snapshots – copy individual table files out of the snapshot
 Manage the backups on an existing MySQL Replication Slave
 mysqlbinlog –read-from-remote-server --raww
 The amount of non-transactional data will continue to shrink in MySQL 5.7


Recap: Backups With Minimal Impact
Limit shared resource usage:
 MEB: --limit-memory, --sleep, --[read|write|process]-threads
 Stream the backup to a remote FS
Backup only as much data as required:
 MEB: Partial backups with --use-tts=with-minimum-locking
 MEB: Skip unused pages with --skip-unused-pages (InnoDB only)
 MEB: Redo log only incremental backups (InnoDB only)

Use 100% non-locking backups:
 MEB: --only-innodb, --only-innodb-with-frm, --no-locking
– Not 100% safe for .frm files, and for non-transactional data


Handling Data Corruption
Insert Picture Here


Backups with Corruption
Sometimes tables get corrupted!
If production is corrupted, but the backup is fine:
 Do a full or partial (--with-tts) restore from the backup
 Apply changes from binary logs

If both the production data, and the backup itself are corrupted:
 InnoDB
– --innodb_force_recovery
– SELECT INTO OUTFILE
 MyISAM
– myisamchk


Backups with Corruption
What to do if:
 You have only full MEB backup, but need to restore single table
 You can not setup spare instance
 You can not use TTS
MyISAM
 Simply copy .frm, .MYI, and .MYD files
InnoDB
 Things get harder
 Each table needs to have the same table ID in the shared tablespace, in
both the backup and production server


Backups with Corruption:
Full Backup and InnoDB
If you can setup separate instance and restore full backup there
 Do it!
 If you can use TTS
– Backup needed table using MEB with option --use-tts
– Restore it


Backups are Corrupted:
If you can not!
 The problem
 Table ID in the InnoDB shared tablespace must be same in the backup and
production server
– They are if:
– this is a backup of the same production server
– you originally cloned this server from this backup and
– You use option –innodb-file-per-table
– Simply copy the .ibd file


If table ID is not the same and you use option –innodb-file_per_table
 Recover manually
– Simulate the internal InnoDB table counter
– Start spare instance
– Create, then drop table with the same structure as the original one
until counter increases to appropriate value
– Import backed up tablespace into spare instance
– Copy .ibd file to production server


But how will you find what to type in CREATE TABLE statement?
 mysqlfrm, the FRM reader will help you
$$ mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333
mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333
## Starting the spawned server on port 3333 ... done.
Starting the spawned server on port 3333 ... done.
## Reading .frm files
Reading .frm files
##
## Reading the city.frm file.
Reading the city.frm file.
#
#
# CREATE statement for city.frm:
# CREATE statement for city.frm:
#
#
CREATE TABLE `test1`.`city` (
CREATE TABLE `test1`.`city` (
…
…


Conclusion
Insert Picture Here


Conclusion
 Backups are a must
 Production constraints are a fact of life
 Difficult problems can be solved with MySQL Enterprise Backup
 MySQL Support is there to help with your specific problems


References
Documentation:
 http://dev.mysql.com/doc/mysql-enterprise-backup/3.9/en/index.html
 http://dev.mysql.com/doc/workbench/en/mysqlfrm.html
 https://blogs.oracle.com/mysqlenterprisebackup/
MySQL Enterprise Backup Trial Downloads:
 https://edelivery.oracle.com/


Graphic Section Divider


Making Backups in Extreme Situations

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Making Backups in Extreme Situations

Similar a Making Backups in Extreme Situations (20)

Más de Sveta Smirnova

Más de Sveta Smirnova (20)

Último

Último (20)

Making Backups in Extreme Situations