Talk from DebConf 2019

1. How MariaDB in Debian uses
Salsa-CI/Gitlab-CI
Ensure smooth upgrades and avoid regressions
In Miniauditório at Tue 23rd July 2019 14:30-15:15
Follow
@ottokekalainen
to
get
link
to
slides!

2. Making use of new Debian infrastructure
Salsa.Debian.org
● Debian’s own Gitlab instance
● First prototype in 2017,
rolled-out big scale in 2018
● See docs at https://wiki.debian.org/Salsa
● Run by Debian Salsa-Admin team
● MariaDB/MySQL packaging
repositories moved from
git.debian.org to
salsa.debian.org in January
2018
Salsa-CI
● Debian packaging specific
Gitlab-CI pipeline
● First version in summer 2018,
very active development all the
time
● See README at
https://salsa.debian.org/salsa-ci-team/pipeline
● Run by Debian Salsa-CI team (I
am a former member of it)
● Repositories for mariadb-10.1,
mariadb-connector-c, galera-3
adopted it in August 2018

3. Salsa-CI is
amazing
If you are a
Debian Developer,
start using it now!
Ensures that on every commit/push:
● Sources build and Debian
packages are generated
● blhc: Build logs don’t have
errors
● reprotest: Builds are
reproducible (no randomness)
● autopkgtests: binaries can run
● lintian: static code analysis
● piuparts: packages install,
uninstall and upgrade

4. ● Stop wasting your time on stupid mistakes and
unnecessary prepare-upload-wait-check cycles.
● Stop wasting Debian archive resources with
excess uploads.
● Give confidence to your contributors who can
run CI on their merge requests.
● Give yourself a peace of mind, reduce stress and
free up time to contribute more to Debian!
● Go from 10x engineer to 20x engineer!

5. Example: debian/gitlab-ci.yml contents in galera-3
Branch: master
include:
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
Branch: buster
include:
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
variables:
RELEASE: 'buster'

6. Example: check Gitlab-CI config path and runner settings

7. Example: salsa.debian.org/mariadb-team/galera-3/pipelines
Gitlab-CI offers real-time logs, graphs,
artifacts download… Everything you need!

8. Example: Commit history and Merge Request statuses

9. Example: Notification email

10. Salsa-CI is very easy for a simple package
See also Augustin Henze’s talk:
debconf19.debconf.org/talks/148-salsa-ci-debian-pipeline-for-developers/

11. ..but MariaDB far from simple

12. 13 million lines of code...

13. One source package generates 25 binary packages...
libmariadb-dev
libmariadbclient-dev
libmariadb-dev-compat
libmariadb3
libmariadbd19
libmariadbd-dev
mariadb-common
mariadb-client-core-10.3
mariadb-client-10.3
mariadb-server-core-10.3
mariadb-server-10.3
mariadb-server
mariadb-client
mariadb-backup
mariadb-plugin-connect
mariadb-plugin-rocksdb
mariadb-plugin-oqgraph
mariadb-plugin-tokudb
mariadb-plugin-mroonga
mariadb-plugin-spider
mariadb-plugin-gssapi-server
mariadb-plugin-gssapi-client
mariadb-plugin-cracklib-password-check
mariadb-test
mariadb-test-data

14. https://packages.debian.org/search?lang=en&suite=all&searc
hon=names&keywords=mariadb-server
https://packages.ubuntu.com/search?lang=en&suite=all&sear
chon=names&keywords=mariadb-server
Multiple upstream versions in maintenance in parallel in
two distros each with multiple releases...

15. The configuration and data needs to persist and work over each upgrade...
Default configuration
/etc/mysql/
/etc/mysql/conf.d
/etc/mysql/conf.d/mysql.cnf
/etc/mysql/conf.d/mysqldump.cnf
/etc/mysql/mariadb.conf.d
/etc/mysql/mariadb.conf.d/50-client.cnf
/etc/mysql/mariadb.conf.d/50-mysql-clients.cnf
/etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf
/etc/mysql/mariadb.conf.d/50-server.cnf
/etc/mysql/my.cnf.fallback
/etc/mysql/my.cnf
/etc/mysql/mariadb.cnf
/etc/mysql/debian-start
/etc/mysql/debian.cnf
Default data
/var/lib/mysql/debian-10.1.flag
/var/lib/mysql/mysql
/var/lib/mysql/mysql/db.frm
/var/lib/mysql/mysql/db.MYI
/var/lib/mysql/mysql/db.MYD
/var/lib/mysql/mysql/host.frm
/var/lib/mysql/mysql/host.MYI
/var/lib/mysql/mysql/host.MYD
/var/lib/mysql/mysql/user.MYI
/var/lib/mysql/mysql/user.MYD
…
/var/lib/mysql/mysql/user.frm
/var/lib/mysql/aria_log_control
/var/lib/mysql/aria_log.00000001
/var/lib/mysql/ibdata1
/var/lib/mysql/ib_logfile1
/var/lib/mysql/ib_logfile0
/var/lib/mysql/performance_schema
/var/lib/mysql/performance_schema/db.opt

16. Lot’s of direct users: 30 % of Debian popcon submitters

17. Hundreds of indirect users and dependencies that need to continue working...
Packages that depend on MariaDB server or client directly
apt-cache rdepends 'default-mysql-s*' 'mariadb-s*' | wc -l
96
apt-cache rdepends 'default-mysql-c*' 'mariadb-c*' | wc -l
73
Packages that depend on MariaDB client libraries
apt-cache rdepends 'default-libmysql*' 'libmariadb*' | wc -l
194

18. MariaDB packaging custom Gitlab-CI pipeline

19. 18 jobs in 5 stages

20. Stage 1: build
Build in target
release plus one older
(ensuring backporting
is still possible).

21. Stage 2: test quality of built packages
Quickly detect if the
software malfunctions
in an obvious way.

22. Stage 3: upgrade from current version in same release
E.g. what will happen to
users when the built
version is uploaded to
unstable (or stable if
maintenance branch).

23. Stage 4: upgrade from the version in (old) stable releases
E.g. what will happen
when users run
dist-upgrade from the
previous stable or
oldstable to the
version being built
now.

24. Stage 5: additional tests
E.g. what will happen if
users upgrade from MySQL
5.7 (which would not
normally happen in a
Debian dist-upgrade).
Also lesser priority
tests that are allowed to
fail.

25. libmysql* to libmariadb* upgrade
# Install all libmysql* available in Debian unstable
- apt-get install -y pkg-config libmysqld-dev libmysqlclient-dev
- pkg-config --list-all
- pkg-config --cflags mysqlclient # mysqlclient.pc from original package
- apt-get install -y ./libmariadb3_*.deb ./mariadb-common_*.deb
- pkg-config --list-all
- apt-get install -y ./libmariadb-dev_*.deb
- pkg-config --list-all
...
- apt-get install -y default-libmysqlclient-dev default-libmysqld-dev
- ldconfig -p # | grep -e mariadb -e mysql
- pkg-config --list-all
- pkg-config --cflags --libs mysqlclient
- pkg-config --cflags --libs libmariadb
- pkg-config --cflags --libs mariadb

26. build mariadbclient consumer Python-MySQLdb
- apt-get install -y pkg-config ./libmariadb-dev*.deb ./libmariadb3_*.deb
./mariadb-common*.deb
- pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with
- apt-get install -y python3-pip
- pip3 install mysqlclient # Compiles module against libmysqlclient
- apt-get purge -y libmariadb-dev # Not needed for run-time
- python3 -c "import MySQLdb; print(MySQLdb.get_client_info())"
Ensures we have a chance to notice if other software that used (legacy)
libmysqlclient and are now automatically using libmariadb3 still works.
Priceless: This caught upstream bugs what otherwise would have gone
unnoticed for a long time since the “damage” is so subtle and invisible.

27. Explore it yourself!
Gitlab, Salsa-CI and MariaDB evolves constantly
salsa.debian.org/mariadb-team/mariadb-10.3

28. Challenges and peculiarities
● Custom gitlab-ci.yml has been
developed now for a little over
a year and grown to 550+ lines.
● Tries to cover all relevant
install/upgrade scenarios.
● Docker images don’t have
systemd running by default,
must activate with
○ sed -i "s/101/0/g" -i
/usr/sbin/policy-rc.d
● Build time max 2h
○ Slowness was tackled with
ccache (and also upstreamed
to Salsa-CI
● Build log max 4 MB
○ build command runs with
■ | tail -n 10000
○ before build starts a a
progress indicator is spawned
■ while true; do sleep 600;
echo "10 minutes passed"
>&2; done &

29. Next additions?
● Fix Jessie to latest upgrade
test that regressied in early
May 2019 with:
○ systemd-udevdstart-stop-daemon:
unrecognized option
'--notify-await'
○ Bug in Jessie or Jessie Docker
image, not MariaDB
● Making RocksDB and TokuDB
plugins build reproducibly
○ Can you help?
● Extend testing of MariaDB
plugins and high-level features
not covered by upstream MariaDB
test suite
● Manipulate config files and
database contents before
starting upgrade to test how
the maintainer scripts can
recover from customized
situations and heterogeneous
environments
● Custom runner guarantee it is
fast?

30. Thank you!
@ottokekalainen
Reminder of next session:
Salsa and Salsa-CI BoF
Room: Miniauditório
Time: Jul 26 (Fri), 14:30