1. KERNEL DEVELOPMENT:
DRAWING LESSONS FROM
"MISTAKES"
Japan Linux Symposium 2009
October 23, 2009
Toshiharu Harada
haradats@nttdata.co.jp
NTT DATA CORPORATION

2. ABSTRACTS
Every kernel developer knows that Linux comes with plenty of precious
documentation as an integral part. From coding style to how to post
patches, almost everything has been documented. However, history
shows that error is human nature. Sometimes developers do not well
know Don’ts, but there are also cases when they make mistakes despite
being aware of such rules. Why this happen is unsolved, but a
documentation, so far missing, of the consequences of this misbehavior
could discourage it. The presenter is project manager of TOMOYO
Linux, a security enhancement feature merged in version 2.6.30.
Thinking open-minded, he decided to share the errors his project
made, wishing it could be a helpful warning to other projects, especially
newcomers. In this presentation, it will try to explain the mistake
circumstances in TOMOYO Linux project, highlighting the thoughts of
project members and the community reactions.

3. “Experience is the name everyone gives to
his mistakes” --- Oscar Wild

4. WHAT’S THIS ALL ABOUT
• Linux comes with a set of great documentation
• DOs and DON’Ts are there already
• Yet we keep on making mistakes
• Finding a missing piece

5. WHAT IS MISSING?
• In my humble opinion:
• Human nature (hard to fix)
• Most of us don’t really like readings (hard to fix)
• Real-life examples taken from TOMOYO Linux project

6. WHO AM I?
• Project manager of TOMOYO Linux

7. OUCH, IS THIS “SECURE LINUX” TALK?
• No! Thistalk has no “security”, no “Policy” or “MAC” (so, please
remain seated)
• If you are interested in them, contact me personally

8. COVERED TOPICS
• Chapter 1:
• Where to find DOs and DON’Ts
• Chapter 2:
• TOMOYO Linux posting history overview
• Chapter 3:
• Step by step introDOs and DON’Ts of the TOMOYO Linux

9. CHAPTER 1
Where to find DOs and DON’Ts

10. Gentle Reminder
Documentation is a part of Linux Kernel
After checking out the kernel, cd to
“Documentation”
Problem is “there are just too many files and
directories” and people prefer coding than
reading

11. How Great is It?
$Documentation/ManagementStyle
“Most people are idiots, and being a
manager means you'll have to deal with it,
and perhaps more importantly, that
_they_ have to deal with _you_.”
“Thing will go wrong, and people want
somebody to blame. Tag you’re it.”

12. Minimal Reading
Entire Scheme
$Documentation/HOWTO
Submitting Patches
$Documentation/SubmitChecklist
$Documentation/CodingStyle

13. References
Note that the title is “How to Participate in
the Linux Community”
Making your code upstream means your
participation in the Linux Community

14. My favorite one
http://www.linuxfoundation.jp/jp_uploads/
seminar20070710/Jon-Dev-Process.pdf

15. CHAPTER 2
TOMOYO Linux by Numbers

16. All Happy Families Resemble Each
Other, Each Unhappy Family Is
Unhappy in Its Own Way. - Leo Tolstoy

17. 3

18. We were 3

19. 3

20. 3rd LSM
module

21. 15

22. Posted 15
times

23. 716

24. Merged since 716
days after the first
post

25. 162

26. Received 162
comments from
LKML

27. Proposal History
http://tomoyo.sourceforge.jp/wiki-e/?JLS2009

28. CHAPTER 3
Drawing Lessons from the “Mistakes”
of TOMOYO Linux Project

29. IN THE 1ST POSTING
I wrote:
All right, that's almost everything. Please
visit the following
URL for the code and documents:
http://tomoyo.sourceforge.jp/wiki-e/
If you want to see the code first, then:
http://tomoyo.sourceforge.jp/cgi-bin/lxr/
source/security/tomoyo/?v=linux-2.6.21.3-
tomoyo-2.0

30. DON’T
Send URL
Send patches

31. WHAT HAPPENED?
•Igot a personal message from Stephen Smalley, a maintainer
of that famous SELinux
–“If you really want feedback or to get your code into the
kernel, you need to do more than post a URL to the code - you
need to break your code down into a number of patches and
post them, just like the AppArmor folks have been doing. “

32. SO WE RUSHED TO POSTED
PATCHES NEXT DAY
• Pavel Machek gave a comment
• “Looks whitespace-damaged to me.”

33. DON’T
Ignore the Linux standard coding style
Always apply checkpatch.pl

34. DO
• Read the $Document/CodingStyle
• Check your code with $scripts/checkpatch.pl
• Also use other $scripts/check*.pl

35. • Jiri Kosina pointed us to make patches bisectable
• “Just a trivial minor nitpick - IMHO this breaks bisectability. It
might be better to add the Kconfig/Makefile patch at the end
of the whole series, so that bisect doesn't end up in the tree in
which Makefile references non-existing files/directories.”

36. DO
• Add the Kconfig/Makefile patch at the end of the whole
series, so that bisect doesn't end up in the tree in which
Makefile references non-existing files/directories.”

37. IN THE 3RD PATCH
• James Morris taught us series of patches should form a thread
• “I'd
also suggest making all of the patches a reply to the first
email, so they can be threaded.”

38. DO
• Send series of patches as children of the first message so that
they can form a thread

39. • James Morris said:
• “Please use standard kernel list handling, per include/linux/list.h”
• YOSHIFUJI Hideaki also mentioned:
• You'reintroducing a custom API, which is open-coded repeatedly
throughout your module.
• All linked lists (at least, new ones) must use the standard kernel
list API.

40. DON’T
Propose new data structure
Use existing one

41. IN THE 5TH POSTING
• James Morris suggested to CC netdev mailing list
• “You should send anything which touches core networking to
netdev, too, and get an ack from one of the core developers
there.”

42. DO
• Carefully choose CCs and get a review from them

43. IN THE 6TH POSTING
• Tetsuoposted 30 series of messages with the subject,
“Subject: [TOMOYO #7 00/30] TOMOYO Linux 1.6.0
released”
• Theproblem was “TOMOYO 1.6.0” did not use LSM and
implemented different hooks

44. DON’T
Try to invent a new API
Respect a standard and follow one

45. • Tetsuoand I knew that posting such patches will never be
accepted
• However, we had been stuck and we couldn’t find another
way
• Our posting was thoughtless, but we were so serious to make
our code upstream

46. IN THE 7TH POSTING
I changed my mind and wrote:
We apologize for the confusion we caused in
the last posting,
but we don't want to give up returning our
work to the mainline.
We cordially request LSM changes to pass
vfsmount parameters.

47. DON’T
Cordially ask to merge your code ;-)
Make good patches and post them

48. • Stephen Smalley kindly responded on the list
–“Don't cordially request it - submit patches to make it happen.
Or work with others who have been submitting such patches. “

49. • Serge E. Hallyn wrote:

50. CONCLUSION

51. • Our mistakes, presented in this slides, are merely the tip of the
iceberg
• We made many failures and we sometimes behaved very
badly
• Nevertheless, this Linux community reacted and even merged
our code

52. • Feedbacks were not limited by mailing list replies
• There were people who sent off-list messages and left advices
in face

53. WITH ALL MY EXPERIENCES
I CAN SAY
• Linux is not just free code
• Linux is great because people are great
• Sending your code is a conversation with people
• Theymight not appear friendly for the first time, but try to
speak them first

54. ACKNOWLEDGMENTS

56. Al Viro, Albert Cahalan, Alexey Dobriyan, Andi Kleen, Andrew
Morton, Bodo Eggert, Casey Schaufler, Chris Wright, Christoph
Hellwig, Crispin Cowan, Daniel Walker, David Howells, David Lang,
David P. Quigley, Greg KH, James Morris, Jamie Lokier, Jiri Kosina,
Jonathan Corbet, Joshua Brindle, KaiGai Kohei, Kamezawa
Hiroyuki, KOSAKI Motohiro, Kyle Moffett, Linus Torvalds, Matthew
Wilcox, Miklos Szeredi, Paul E. McKenney, Paul Moore, Pavel
Machek, Peter Dolding, Peter Zijlstra, Rik van Riel, Serge E. Hallyn,
Seth Arnorld, Shaya Potter, Stephen Smalley, Tim Bird, Trond
Myklebust, Valdis Kletnieks, William Leibzon, YOSHIFUJI Hideaki

57. WE COULDN’T HAVE DONE IT
WITHOUT YOUR HELP
Al Viro, Albert Cahalan, Alexey Dobriyan, Andi Kleen, Andrew
Morton, Bodo Eggert, Casey Schaufler, Chris Wright, Christoph
Hellwig, Crispin Cowan, Daniel Walker, David Howells, David Lang,
David P. Quigley, Greg KH, James Morris, Jamie Lokier, Jiri Kosina,
Jonathan Corbet, Joshua Brindle, KaiGai Kohei, Kamezawa
Hiroyuki, KOSAKI Motohiro, Kyle Moffett, Linus Torvalds, Matthew
Wilcox, Miklos Szeredi, Paul E. McKenney, Paul Moore, Pavel
Machek, Peter Dolding, Peter Zijlstra, Rik van Riel, Serge E. Hallyn,
Seth Arnorld, Shaya Potter, Stephen Smalley, Tim Bird, Trond
Myklebust, Valdis Kletnieks, William Leibzon, YOSHIFUJI Hideaki

58. AND OF COURSE
• We thank NTT DATA CORPORATION for their support of
the project

59. RELATED PRESENTATIONS
Realities of Mainlining
- Case of the TOMOYO Linux Project - Time to Glean
!#$%'$()*+,-$.)/0'1$2*3$0.4$%+0+'4
Toshiharu Harada
haradats@nttdata.co.jp July 25, 2008
haradats@gmail.com Toshiharu Harada
haradats@nttdata.co.jp
Kentaro Takeda
NTT DATA CORPORATION Tetsuo Handa
NTT DATA CORPORATION
July 9, 2008

60. What does it mean being an
What does it mean being an
Open Source project manager Open Source project manager
in Enterprise in Enterprise
enterprise edition Open Source Edition
LinuxCon 2009 (Business) LinuxCon2009 (Business)
September 23, 2009
Toshiharu Harada
September 23, 2009
haradats@nttdata.co.jp
Toshiharu Harada
NTT DATA CORPORATION haradats@gmail.com
TOMOYO Linux Project

61. TRADEMARKS
• Linuxis a registered trademark of Linus Torvalds in Japan and
other countries
• TOMOYOis a registered trademark of NTT DATA
CORPORATION in Japan