1. Introduction
Continuous Integration
Epilogue
Glimpse into CI testing
Linux Conf Au 2014: Continuous Integration Miniconf
Raghavendra Prabhu
raghavendra.prabhu@percona.com
Percona
6th January, 2013
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

2. Introduction
Continuous Integration
Epilogue
Outline
Introduction
Continuous Integration
Epilogue
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

3. Introduction
Continuous Integration
Epilogue
Overview
Why CI test
More free time for developers! https://xkcd.com/303/
PXC and Galera
MySQL testing framework
Re-use
Newer approaches
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

4. Introduction
Continuous Integration
Epilogue
Jenkins
Trigger
From VCS to packages
Manual v/s Polling
Dependencies and blocking
Workflow
Parameterized triggers
VCS clones
Build plumbing and fork-join
Works well with copying artifacts
Matrix builds
Selective rebuilds
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

5. Introduction
Continuous Integration
Epilogue
Sysbench
Simultaneous dispatch to all nodes
Re-use of MTR.
Benchmarking
History and Comparison
Code regressions
Testing
Latent bugs
Seen only at high load
Synchronous replication
Transaction commits/rollbacks and BF aborts
Latency instrumentation
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

6. Introduction
Continuous Integration
Epilogue
RQG
Combinations testing
Pruning of infeasible
Astronomically large number
Corner cases
Bugs hide
Stress
Degrees
mm, ms, mmm and more!
Locking bugs
DDL and MDL
Other locks
InnoDB Row deadlocks
Hung tests
Server deadlock
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

7. Introduction
Continuous Integration
Epilogue
End-to-End testing
Also works as a smoke test
Snapshot State Transfer
Re-uses PXB test suite with Xtrabackup
Critical component
Newer nodes are cloned
Flow
start node1 => load => start node2 => checksum
Configuration
Tests different platforms
13 supported ones
Outcome
Ensures PXC works as a cluster as intended
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

8. Introduction
Continuous Integration
Epilogue
Replication testing
Overlaps with other tests in coverage
Tests replication between major versions
Intended to test rolling upgrades/downgrades
Re-uses earlier tests - MTR, sysbench, SST
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

9. Introduction
Continuous Integration
Epilogue
Miscellaneous
Techniques found useful
lock_wait_timeout to expose MDL bugs
Use of both release and UNIV_DEBUG builds
Doubles the number of platforms
Debug ones hit earlier assertions
Release
Go further
Hit load-related issues
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

10. Introduction
Continuous Integration
Epilogue
Future
Testing at higher level
Using Chef, Puppet
Mainly for testing packaging
Also to test distro idiosyncrasies
Automated handling of test results
Extraction of the backtraces etc.
Bug reporting!
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

11. Introduction
Continuous Integration
Epilogue
Conclusion
Who says testing can’t be fun!
http://jenkins.percona.com/user/
raghavendra-prabhu/my-views/view/PXC/
Degree of overlap among tests.
Having multiple platforms help
Others: bzr, launchpad, IRC, git(?)
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into

12. Introduction
Continuous Integration
Epilogue
Me
Raghavendra Prabhu
http://wnohang.net/about
Software Engineer at Percona and Product Lead of PXC
https://launchpad.net/percona-xtradb-cluster
Raghavendra Prabhu
Percona XtraDB Cluster before every release: Glimpse into