PR Karthik sharing challenges faced while migrating data to the Open Source Platform and the solutions his team employed to address the issues.

1. Customer Experience
InnoDB Cluster Implementation
Presenter
Karthik P R
Founder Mydbops
www.mydbops.com info@mydbops.com

2. About Mydbops
● Founded in 2015, HQ in Bangalore India with 150+ customer base across the globe.
● Mydbops is on Database Consulting with core specialization on MySQL and MongoDB Administration and
Support.
● We have expert team with 20+ certified DBA’s providing full time support and currently managing 300+
servers.
● Mydbops was created with a motto of developing a DevOPS model for Database administration offering
24*7 expert remote DBA support.
● We help organisations to architect and scale systems in MySQL/Mongo by implementing the advanced
technologies in industry which are completely open source.

3. Mydbops is into MySQL/MongoDB Support and Consulting. It is founded by experts
who have scaled database at Yahoo! ,Percona and Datavail. We are providing an
expert level support and 24*7 monitoring for MySQL databases and its related
technologies like MariaDB , Percona ( also clustering ) . We support modern
database technologies in MySQL which includes Galera ( Clustering ), Group
Replication , SQL aware Load balancers like Maxscale / ProxySQL.
About Mydbops

4. mysqlsupport@mydbops.com
Our Clients
www.mydbops.com

5. CEO / DB Architect
About Me

6. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents

7. ● Group Replication Library
● Virtually Synchronous replication (MySQL 5.7 )
● GCS is based on Paxos ( Variant of Mencius )
● High Availability solution on InnoDB
● Consistent data across nodes
● Shared Nothing Cluster
● Supported on all Platforms
● Inbuilt in MySQL ( No third party plugin )
Introduction : InnoDB Cluster

8. Introduction : InnoDB Cluster
MySQL Router
MySQL Shell
MySQL Group Replication

9. Transaction in Group Replication
Transaction in Group Replication

10. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents

11. A largest retail store in USA makes a migration to Open source Platform.
● Migration to Cloud ( Google Cloud )
● Monolithic to Microservice Platform
● Application servers on Containers ( Dockers )
● MySQL and MongoDB as backend data stores.
● Redis as a caching layer
● Consul for service discovery and Ansible for Config management.
Project Background

12. MySQL Info
● Automated Deployment with MySQL Shell ( In GCC)
● MySQL Enterprise servers.
● Virtual Environment
● Single Primary Mode cluster
● MySQL Version 5.7.17 ( started with )
● No read / write Split up
Project Background

13. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents

14. Case 1) Heavy Data loading.
Initial Loading of CSV files of heavy size breaks
the cluster and rollback the transaction.
Becomes our first blocker as data validating can
be made once the data is loaded.
Challenges Faced

15. Solutions Proposed.
● Stop the group replication reset the Master GTID’s and make the sync back in async mode ( normal
replication )
● Use a hot backup solution like MySQL Enterprise backup and rebuilt it.
The Solution 1 was comparatively faster in comparing the data set we had.
Challenges Faced

16. Case 2 ) Setting up HA for MySQL Router.
MySQL router can become a single point of failure
Challenges Faced

17. Solutions Proposed.
Implement Router as a layer with docker containers as
a part of application deployment and maintain the config
repository based on microservices.
Challenges Faced
MR MR MR MR MR

18. Case 3: MySQL bugs affected the Stress tests
MySQL Version : 5.7.17
Replication: In previous versions issuing STOP GROUP_REPLICATION stopped the plugin but the server
still accepted transactions. This meant the transactions were not transmitted to the group. To make STOP
GROUP_REPLICATION safer, now super_read_only is set to ON immediately upon issuingSTOP
GROUP_REPLICATION, which ensures no transactions are accepted. (Bug #25495067, Bug #84795)
Replication: Interleaved transactions could sometimes deadlock the slave applier when the transaction
isolation level was set to REPEATABLE-READ. (Bug #25040331)
Challenges Faced

19. Solutions Proposed.
MySQL Version : Those bugs were fixed in MySQL 5.7.20 and MySQL 5.7.19 respectively. We made
MySQL 5.7.20 as a standard in our repository for production.
Note :
Group_replication_transaction_size_limit was added in MySQL 5.7.19
Challenges Faced

20. Case 4: Isolation Level and Open transaction
Repeatable read can be performance killer in a cluster ( optimistic locking ). And open transactions from
application or users make it much worse.
Solution Proposed :
Read Committed was made as a standard isolation level and open transaction alerts based on
performance_schema was set to find application code bugs or user level
Challenges Faced

21. Case 5: Schema changes
The heavy DDL can block the parallelism in the cluster and can make the secondary to RECOVERY
state.Even INNODB Online alter can't help much as it needs a table rebuild.
Solution Proposed :
Third party Tools can be saviour - Percona toolkit or Gh-OST ( can be )
Challenges Faced

22. Case 6 : InnoDB cluster Can’t rejoin .
When the node is rebuild using MEB , the Group Replication is running fine. But the InnoDB cluster can’t
find the right UUID and marks the cluster as failed node. Can be validated via MySQL Shell
select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | b0b1603f-83ef-11e6-85a6-080027de0e0e | box01 | 3306 | ONLINE |
| group_replication_applier | bb29750c-83ef-11e6-8b4f-080027de0e0e | box02 | 3306 | ONLINE |
| group_replication_applier | bbu3761b-83ef-11e6-894c-080027de0t0e | box03 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
Challenges Faced

23. mysql> show tables;
+-----------------------------------------+
| Tables_in_mysql_innodb_cluster_metadata |
+-----------------------------------------+
| clusters |
| hosts |
| instances |
| replicasets |
| schema_version |
+-----------------------------------------+
5 rows in set (0,00 sec)
Solution and fix ( Temporary Patch ).
The cluster metadata table has to be updated manually post a rebuild if there is a mismatch in UUID.
Challenges Faced

24. 1. Data loading
2. Setting up HA for MySQL Router.
3. MySQL and Router Version.
4. Isolation level and open transaction
5. Handling the DDL’s
6. DB Recovery and InnoDB Cluster.
Challenges Faced

25. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents

26. ● Read Write Split up from App / ProxySQL
● Planning to Upgrade MySQL 5.7.22 or MySQL 8.0
● Migration to Bare Metal server.
● Single Click Upgrades on Ansible
Future Enhancement

27. Community Sponsorship

28. Thank You !!!