MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.

$ whoami

Miguel Araújo
• MySQL Developer since 2011
• AdminAPI Tech Lead @ MySQL Shell team
• Living in Portugal
• http://about.me/miguel-araujo
• @M1guelAraujo

MySQL Shell AdminAPI

MySQL Shell
• Interactive multi-language console interface that supports
development and administration for the MySQL Server:
• Supports JavaScript, Python and SQL
• Interactive and batch operations (scripting)
• Supports Document and Relational models (X and MySQL protocols)

MySQL Shell
• Exposes full Development and Administration APIs:
• Scriptable “DevOps” APIS
• Unified interface for developers and DBAs

MySQL Shell AdminAPI
• Creation and Management of InnoDB Clusters / ReplicaSets
• Available in both JavaScript and Python; and command-line
• Simple and straight-forward
• Flexible, powerful and secure!

•MySQL Group Replication
•MySQL Shell
•MySQL Router
InnoDB Cluster
“A single product – MySQL – with high
availability and scaling features baked in;
providing an integrated end-to-end
solution that is easy to use.”

•MySQL Replication
•MySQL Shell
•MySQL Router
“We take a strong technology that is very
powerful but can be complex, and provide
an easy-to-use AdminAPI for it in the
MySQL Shell”
InnoDB ReplicaSet

Creating a Cluster - AdminAPI
// Configure Instances for Cluster usage
mysqlsh-js> dba.configureInstance(“root@mysql1:3306”)
// Create the Cluster
mysqlsh-js> c root@mysql1:3306
mysqlsh-js> cluster = dba.createCluster(“myCluster”)
// Add Instances to the Cluster
mysqlsh-js> cluster.addInstance(“root@mysql2:3306”)

Creating a ReplicaSet- AdminAPI
// Configure Instances for ReplicaSet usage
mysqlsh-js> dba.configureReplicaSetInstance(“root@mysql1:3306”)
// Create the ReplicaSet
mysqlsh-js> c root@mysql1:3306
mysqlsh-js> replicaset = dba.createReplicaSet(“myReplicaSet”)
// Add Instances to the ReplicaSet
mysqlsh-js> replicaset.addInstance(“mysql2:3306”)

Couldn’t be easier!
Powerful solutions supported by Shell’s ease of use.

Node Provisioning

Provisioning
• Provisioning a server is the process of configuring and getting it
ready to be used in an InnoDB Cluster/ReplicaSet.
• In InnoDB Cluster, the server needs to have all clusters’ data to be
able to join it
• That’s typically done by:
1. Performing a backup of one of the cluster members
2. Restoring the image in the new member that wants to join

Distributed Recovery
• Distributed recovery is a process that runs when a new member is
added to the group and has missing transactions
• A donor that can provide the missing data is automatically selected
• The catch-up happens transparently and the server is
synchronized with the rest of the cluster

Incremental Recovery to Provision
• Incremental recovery fetches only the missing data based on the
GTIDs into
- The cluster must be able to provide all the missing data
• Relies on the classic Asynchronous Replication
• Also provides group transactional information

Some data is inserted in
the DB

The data modifications
are stored in the binlog.
And the transactions are
delivered to every cluster
member

Group Replication is
eventually synchronous.
Guarantees that
transactions are delivered
to all members in the
same order.

The binlogs keep track of
all data modifications.

A new member wants to
join the cluster

A new member wants to
join the cluster
The missing data is
fetched and the server
synchronizes with the rest
of the cluster

A donor is automatically
selected to fetch the
missing data from, using
plain Asynchronous
Replication

The instance successfully
joins the cluster!

May take a considerable amount of time !

• But what if the cluster members lost all binary logs?
• Or even... Just some data is missing that is not
available on any binary log
• What about space constraints?
?

Some binary logs were
purged.
And the instance has no
data yet.

purged.
data yet.
Failed attempt to join the
cluster !

MySQL Clone Plugin
• Performs physical snapshots of databases
Really fast!
• Transfers snapshots over the network to provision
servers
Can be used remotely!
• Replaces all server data
Available in InnoDB
Cluster and ReplicaSet !

Clone to Provision
Common issues solved with Clone:
• Very large datasesets
• Incomplete binary log history
• Re-provision diverged instances

Clone to Provision
purged.
data yet.

Clone to Provision
purged.
data yet.
Clone required !

Clone to Provision
A donor is automatically
selected.
And a full snapshot of its
data is performed.

Clone to Provision
The instance successfully
joins the cluster!

Clone-based
Recovery Live Demo
InnoDB Cluster

•Incremental recovery is possible if:
• At least 1 member can provide the
joiner with the same set of transactions
it has processed.
• The joiner does not have errant
transactions
•Incremental recovery appropriate if:
• No purged transactions
• Joiner does not have an empty GTID
set
• GTID support and binary logs were
always enabled
Which recovery shall be used?

Incremental recovery safe and appropriate:
AdminAPI Guidance

AdminAPI Guidance
Clone-based recovery required:

Clone-based
Recovery Live Demo
InnoDB ReplicaSet

InnoDB Cluster/ReplicaSet
• Built-in support for full instance provisioning!
• AdminAPI transparently:
ü Setups the cluster/replicaset to be ready for clone usage (plugin setup, user grants, etc.)
ü Detects every possible scenario and guides the user accordingly to the best provisioning
method
ü Still allowing manual intervention to choose the provisioning method

Monitoring

Node Provisioning Monitoring
• The AdminAPI was extended to provide monitoring features:
ü For any type of recovery method: clone or incremental
ü Real-time progress information
ü Detailed stage status
ü Asynchronous progress information from cluster.status()

Node Provisioning Monitoring
Clone-based recovery
progress information from:
cluster.status()

Cluster Status
Entry point for Cluster
monitoring:
ü Provides general info about the
cluster
ü Provides info about each member
status
mysqlsh-js> cluster.status()

Extended Status
Provides information of:
• GR Group name + protocol version
• Cluster members UUIDs
• Cluster members roles and states
• Read-only settings
mysqlsh-js>
cluster.status({extended: 1})

Extended Status
• Transactions processed by
connection and applier threads
mysqlsh-js>

Extended Status
• Detailed stats about the replication
machinery of each cluster member
mysqlsh-js>

ReplicaSet Status
Entry point for ReplicaSet
monitoring:
ü Provides general info about the
replicaset
ü Provides info about each member
status
mysqlsh-js> replicaset.status()

Router Monitoring
• Router info: name, version
• Last check-in
• Hostname
• RO/RW end ports
mysqlsh-js> cluster.listRouters()

Maintenance and
Troubleshooting

Anything that can go wrong will go wrong.”“

Monitoring a Cluster/ReplicaSet can provide info/hints about:
• Faulty Networks
• Server crashes/failures
• High latencies
Maintenance
• Total outages
• … and much more!

Cluster Options
• The cluster global configuration options
• Each cluster member settings
mysqlsh-js> cluster.options()

memberWeight
Next primary in line when a
failover happens
autoRejoinTries
Automatically join back group
after leaving/being expelled
expelTimeout
Avoid frequent member expels due
to less reliable networks
consistency
Transaction consistency
guarantees
Cluster Options

memberWeight
Highest value will be first to
be elected as primary in a
failover
autoRejoinTries
Set to 3
expelTimeout
Set to 5
Cluster Options – Best practices

Dynamically Changing Settings
// Change the member-weight value of a member
mysqlsh-js> cluster.setInstanceOption(“mysql2:3307”, “memberWeight”, 75)
// Change the auto-rejoin-tries of the cluster
mysqlsh-js> cluster.setOption(“autoRejoinTries”, 3)
// Change the cluster’s consistency level
mysqlsh-js> cluster.setOption(“consistency”, “AFTER”)

Topology Modes
• The cluster’s primary instance can be changed, immediately:

Topology Modes
• The topology-mode can be changed live:
// Change the topology-mode to multi-primary
mysqlsh-js> cluster.switchToMultiPrimaryMode()
// Change back to single-primary-mode
mysqlsh-js> cluster.switchToSinglePrimaryMode()

Security
• Internal recovery accounts have randomly secure generated
passwords
• But security policies may enforce passwords lifetime
ü Support to reset a cluster’s recovery accounts credentials added
// Reset the recovery accounts credentials
mysqlsh-js> cluster.resetRecoveryAccountsPassword()

Some scenarios may require intervention…
• Network partitions
• Complete outage
• Server upgrades
Troubleshooting
• Crashed members of a Cluster
or ReplicaSet

Network Partitions
• A network partition can cause the loss
of quorum
• The cluster is not able to process any
more transactions

Network Partitions
• Using a single AdminAPI command
the cluster can be easily unblocked
mysqlsh-js>
cluster.forceQuorumUsingPartitionOf(“clusterAdmin@server1:3306”)

Complete Outage
• Very unlikely to happen!
• But...
• When it happens, restoring the instances back online is not enough
• The Group Replication group is completely gone
• Other configurations in place may have been lost

Complete Outage
• Easily restore the Cluster back to its previously configured state
using:
mysqlsh-js> c clusterAdmin@server1:3306
mysqlsh-js> dba.rebootClusterFromCompleteOutage()

Rejoin Instance
• An instance may not be able to automatically rejoin the
cluster/replicaset:
• Instance running MySQL 5.7 without settings persisted
• Instance from minority partition
• Invalidated primary from ReplicaSet

Rejoin Instance
• Full detection of best provisioning method supported for rejoining
instances
mysqlsh-js> cluster.rejoinInstance(“server3:3306”)
mysqlsh-js> replicaset.rejoinInstance(“server3:3306”,
{recoveryMethod: “clone”})

Upgrade Checker
• Verify whether instances are ready or not for upgrading to
MySQL 8
• Available through the objectutil
mysqlsh-js> util.checkForServerUpgrade(“root@localhost:3306”)

Upgrade Checker

Closing

Final Thoughts
• InnoDB Cluster is THE built-in HA solution for MySQL
• InnoDB ReplicaSet fully integrates MySQL Replication with Router
and Shell brings ease of use!
• MySQL Clone Plugin pushed InnoDB Cluster and ReplicaSet to a
whole new level of usability!
• MySQL Shell is the unified interface for developers and DBAs and
the frontend manager for InnoDB Cluster and ReplicaSet

MOAR Info
• https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-
cluster-userguide.html (InnoDB cluster userguide)
https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/ (Shell APIs JS Ref manual)
https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/ (Shell APIs Python Ref manual)
• https://mysqlhighavailability.com (Blogging)
• http://mysqlserverteam.com/ (Blogging)
• https://mysqlcommunity.slack.com (Community Slack)

Thank you!

MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie

Similar to MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie (20)

More from Miguel Araújo

More from Miguel Araújo (13)

Recently uploaded

Recently uploaded (20)

MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie