Adding CI for the kernel is not easy, but the Continuous Kernel Integration (CKI) team is aiming to do just that.

1. Continuous Kernel
Integration
Major Hayden
@majorhayden
June 1, 2019 - Texas Linux Fest
Photo: David Mark

2. Major Hayden
Principal Software Engineer, Red Hat
major@redhat.com
🤖 Kernel testing automator (CKI Project)
😺 Maintainer of icanhazip.com
📻 Ham radio operator (W5WUT)
🤦‍♂ Hoarder of domain names

3. It all started with a hunt for
a new challenge.
Photo: NASA

4. JOB INTERVIEW:
“So we want you to help us
do CI for the Linux kernel.”
Photo: NASA

5. The team was moving quickly
and I had a lot to learn.
Photo: NASA

6. As it turns out,
the Linux kernel is complicated.
This creates lots of challenges.

7. CHALLENGE:
It moves really quickly.

8. ≈ 14,000 changesets per release
from over 1,700 different developers
≈ 150-300 are first time contributors
Source: LWN

9. kubernetes, a very active project,
has 45,264 changesets since 2014
(that’s equivalent to about three Linux kernel releases)
Source: Stackalytics Photo: CNCF

10. From 5.0 to 5.1:
545,000 lines added
289,000 lines removed
(net growth ≈ 256K lines)
Source: LWN

11. CHALLENGE:
The kernel development
process is complex.
Source: LWN

12. “If it’s not on the mailing list,
then it didn’t happen.”
Source: LWN

13. CHALLENGE:
Merging a patchset
into the kernel is not the same
as a simple pull request.

14. Lifecycle of a
kernel patch
2018-02-23: Submitted a patch for usb-serial
2018-02-26: Merged by subsystem maintainer
2018-02-28: Added to linux-next
2018-04-08: Released in stable 4.16.1 and 4.14.33
2018-06-03: Released in mainline 4.17
2018-10-21: Released in stable 3.16.60
9608e5c0f079 USB: serial: ftdi_sio: add RT Systems VX-8 cable

15. Linux
does a
lot of
stuff
that is
difficult
to test
Photo: Wikipedia

16. When I talk about doing CI for the Linux kernel:
“Should we really be doing that?”
Source: LWN

17. Much more of the work involves
winning over hearts and minds
rather than writing good code.
Source: LWN

18. Initial reaction:
Source: LWN

19. After thinking it over for a little while:
Source: LWN

20. OUR MISSION STATEMENT:
Find bugs before they are
merged and prevent
them from being merged.

21. INITIAL GOALS:
1. Verify patchset merges cleanly
2. Ensure the kernel compiles
3. Boot the kernel and run LTP
4. Send a report to people who care
Photo: pxhereLTP = Linux Test Project

22. 🐣
SUBSYSTEM*
🔥
LINUX-NEXT
🐧
MAINLINE
🐿
STABLE
🎉
RHEL KERNEL
Simplified Kernel patch flow
* There may be multiple layers of subsystem trees in some areas of the kernel, especially networking.
🤔
DEVELOPER

23. 🐣
SUBSYSTEM*
🔥
LINUX-NEXT
🐧
MAINLINE
🐿
STABLE
🎉
RHEL KERNEL
Where we are now
* There may be multiple layers of subsystem trees in some areas of the kernel, especially networking.
🤔
DEVELOPER
🛃
🛃
🛃

24. 🐣
SUBSYSTEM*
🔥
LINUX-NEXT
🐧
MAINLINE
🐿
STABLE
🎉
RHEL KERNEL
Where we want to be
* There may be multiple layers of subsystem trees in some areas of the kernel, especially networking.
🤔
DEVELOPER
🛃
🛃
🛃
🛃

25. Time to talk infrastructure

26. FIRST ITERATION:
Jenkins pipeline on a single server
Lots of CPU contention from kernel compiles (24 CPUs)
We used the same gcc/glibc toolchain for everything
Jenkins upgrades were often problematic
Constantly reverting plugin upgrades

27. SECOND ITERATION:
Jenkins pipeline with OpenShift
Much easier to scale kernel compiles (128 CPUs)
Each container had the right compiler toolchains for its compile job
Jenkins upgrades and plugin upgrades were still rough
Worker containers needed lots of extra Java components to talk to Jenkins

28. THIRD ITERATION:
Self-hosted GitLab with OpenShift
Tightly integrated repositories and CI with easy, controlled scaling
GitLab runner containers could be stripped down to basic components
We added integration testing for the pipeline code itself
Easy runner migration to other OpenShift/Kubernetes deployments
Access to 768 CPUs

30. We start testing kernels when
certain events occur:

31. PIPELINE TRIGGER KERNEL REPO
Baseline tests
Checkout the current HEAD of the kernel tree and
trigger GitLab pipelines to compile the kernel and run tests.

32. PIPELINE TRIGGER PATCHWORK
Patchset tests
Check for new patches in Patchwork
and start testing pipelines in GitLab

33. PIPELINE TRIGGER
STABLE
PATCH QUEUE
Stable patch queue tests
Check for new patches the stable kernel patch queue
and start testing pipelines in GitLab

34. Kernel test pipeline
MERGE
COMPILE
(MULTI-ARCH)
NATIVE ARCH
BOOT/TESTING
LINT
(INTERNAL)
REPORTING
🤔
MAILING LIST
GITLAB RUNNER

35. Native architecture testing
Kernels are booted on their native architecture using Beaker
Beaker manages the pool of machines, runs tests, and provides results
44 kernel subsystem tests are open source (more are coming!)
We test a few different architectures...
Photo: Jonathan Cutrer

36. Photo: Wikipedia
aarch64

37. ppc64 and ppc64le
(IBM POWER)
Photo: common.org

38. s390x
(IBM Z)
Photo: Wikipedia

39. x86_64
Photo: Wikipedia

40. CKI Statistics
Over 100K
individual tests per month
Over 11K
GitLab pipelines run
Under 6 minutes
to compile a kernel
for 4 architectures
Under 2 hours
between patch proposal
and feedback sent (stable)

41. The best part is that we are not
alone in this endeavor.

42. Upstream
involvement in
automated
kernel testing
is growing
0day (Intel)
CKI Project (Red Hat)
Fuego (Sony)
kernelci.org (Linux Foundation)
Linaro
Patchwork
syzkaller/syzbot (Google)
Yocto project (Linux Foundation)

43. How can I help?

44. Read our blog:
https://cki-project.org/

45. Contribute to our tests!
https://github.com/cki-project
https://gitlab.com/cki-project

46. Join us at the kernel CI
hackfest in Lisbon, Portugal
after Linux Plumbers 2019!
Photo: Wikipedia

47. So what’s with this
background?

48. It’s a photo from the
Curiosity Rover on Mars.

49. It’s a photo from the
Curiosity Rover on Mars.
EARTH

50. It’s also a metaphor for
how much work still
remains to be done with
kernel testing.
HOW MUCH
WE’VE DONE
SO FAR

51. Thank you!
Major Hayden
@majorhayden
June 1, 2019 - Texas Linux Fest
Photo: David Mark