Talk given at NCrafts 2018 conference (https://ncrafts.io/)

1. Arnaud Porterie - @icecrime
Building software: the lessons
from Open Source
Arnaud Porterie - @icecrime
VP Engineering - Vente Privée

2. Open source matters
Nobody can afford to ignore the open source movement
Virtually every company uses open source infrastructure software
“Software is eating the world, open source is eating software”

3. Kubernetes
# of commits per month - 6 months moving average

4. Kafka
# of commits per month - 6 months moving average

5. Linux
# of commits per month - 6 months moving average

6. The open source world seems
to be doing something right

7. Leveraging open source code in your company
Contributing to open source as a company
Open sourcing company’s code
What this talk is not about

8. What this talk is about
Taking inspiration from open source in the way we
build software in-house software

9. Alternative title
How my role as VP Engineering today is
influenced by my previous experience
managing a large scale open source project
(they said it was too long ¯_(ツ)_/¯)

10. Arnaud Porterie - @icecrime
A primer on open source
project organization

11. Users
● File issues, report bugs, ask for features
● Influence the project by voicing their opinions
Participants in an Open Source project

12. Contributors
● Send patches, fix bugs, write documentation
● Often are also users of the project
● Influence the project by submitting modifications
Participants in an Open Source project

13. Maintainers
● Guardians of the temple
○ Responsible for the overall health of the project
○ Quality of the code
○ Project scalability
○ Project testing and release cycle
○ Culture of the community
● Typically are also contributors of the project
● Influence the project by deciding what goes in
Participants in an Open Source project

14. ● Read the contributing guide (usually CONTRIBUTING.md)
● Discuss with maintainers
○ Explain the need
○ Explain how you intend to implement it
● Produce the contribution
● Submit a Pull Request (GitHub)
● Patch gets merged when two maintainers give their LGTM
Typical contribution process

15. Upstreams Downstreams
Managing dependencies
Code flow

16. In Open Source, your dependencies (upstreams) don’t work for you
● What you need to be done you contribute yourself
As a maintainer, you play nice with those who depend on you (downstreams)
● When they come in to file an issue or contribute a patch, be helpful
Managing dependencies

17. Arnaud Porterie - @icecrime
The elements of healthy
open source

18. Healthy open source
An open codebase with identified maintainers is not enough
● A healthy open source community fosters collective intelligence
● Not a coincidence but a deliberate act of designing a community
Under the right circumstances, groups are remarkably intelligent, and are
often smarter than the smartest people in them.
The Wisdom of Crowds, James Surowiecki

19. Patterns of a wise crowd
Culture & Empire - Pieter Hintjens
ISBN-13: 978-1492999775

20. Patterns of a wise crowd

21. Patterns of a wise crowd
● Strong mission: a formulation of the single main problem to solve
● Free-entry: strangers may walk in and get involved
● Strong protocols: a set of rules to collaborate properly
● Fair authority: a scalable authority to enforce the rules
● Transparency: all activity takes place in the open
● …

22. Patterns of a wise crowd
Successful online communities expose those patterns
For an open source project, it’s a matter of survival

23. Impact of wise crowds patterns
Ignoring wise crowds patterns Applying wise crowds patterns
Open source software
Pushes the community away
Will inevitably die
Scales to thousands contributors
Maintains quality and velocity
Has extreme resilience

24. Impact of wise crowds patterns
Ignoring wise crowds patterns Applying wise crowds patterns
Open source software
Pushes the community away
Will inevitably die
Scales to thousands contributors
Maintains quality and velocity
Has extreme resilience
Closed source software
? ?

25. Arnaud Porterie - @icecrime
Looking back at the enterprise

26. Is the enterprise enabling wise crowds?
● No universal answer: there are as many organizations as there are companies
● The traditional enterprise optimizes for control and predictability
○ Contrary to open source, applying the patterns is not a matter of survival
● Some of the common organizational pitfalls are clear wise crowds antipatterns
with immediate impact on the software it produces

27. “Application teams” commonly yield
● Lack of a strong mission expressed in user terms
● Arcane undocumented maintenance knowledge
Wise crowds antipatterns
Wise crowds patterns
Strong mission Measurable success
Free entry High scoring
Transparency Decentralization
Full remixability Free workspaces
Strong protocols Smooth learning
Fair authority Regular structure
Non-tribalism Positivity
Self-organization Sense of humor
Tolerance Minimalism

28. Organizational silos commonly yield
● Strong sense of ownership
● Membership model / “us and them” situations
● Heterogeneous structures between departments
Wise crowds antipatterns
Wise crowds patterns
Strong mission Measurable success
Free entry High scoring
Transparency Decentralization
Full remixability Free workspaces
Strong protocols Smooth learning
Fair authority Regular structure
Non-tribalism Positivity
Self-organization Sense of humor
Tolerance Minimalism

29. Top-down decision making commonly yields
● Complexity (decision decoupled from constraints)
● Suboptimal, insufficiently challenged solutions
● Centralization and an inability to scale
● Reluctance to express conflicting opinions
Wise crowds antipatterns
Wise crowds patterns
Strong mission Measurable success
Free entry High scoring
Transparency Decentralization
Full remixability Free workspaces
Strong protocols Smooth learning
Fair authority Regular structure
Non-tribalism Positivity
Self-organization Sense of humor
Tolerance Minimalism

30. Impact of wise crowds patterns
Ignoring wise crowds patterns Applying wise crowds patterns
Open source software
Pushes the community away
Will inevitably die
Scales to thousands contributors
Maintains quality and velocity
Has extreme resilience
Closed source software
Relies on tribal knowledge
Produces lower quality output
Tends to build castles over cities ?

31. Can we raise the “collective intelligence score”
of an enterprise by importing patterns of the
open source world?

32. Arnaud Porterie - @icecrime
Inner source & practical
implementation

33. Inner source
https://en.wikipedia.org/wiki/Inner_source
Inner source is the use of open source software development best practices
and the establishment of an open source-like culture within organizations.
The organization may still develop proprietary software, but internally opens
up its development. The term was coined by Tim O'Reilly in 2000.

34. Impact on teams dynamics
Distinguishing maintainer and contributor roles
● Alters the sense of ownership
○ Maintainers “own” the quality, coherence, and sustainability of the project
○ The organization collectively “owns” the code
● Contributors becomes an open group (free entry principle)
○ Anyone can walk in and participate
● Maintainers also becomes an open group! (fair authority principle)
○ Constructive participation to the project must be rewarded

35. Impact on teams dynamics
Encouraging cross-teams contributions
● Puts the collective goal above individual ownership considerations
● Raises the bar for best practices, particularly on documentation and testing
● Gives individual teams more control of their destiny
● Keeps the development effort where the need exists versus where its implemented

36. ADOPTION
Other engineering teams start
depending on the project
CONTRIBUTIONS
Downstreams participate in
the project’s development
PROJECT’S TEAM GROWTH
The project promotes maintainers to
follow and sustain a healthy growth
Grow software, not teams

37. Impact on teams dynamics
More importantly
● Creates a community where people collaborate and learn from each other
● Creates a environment where engineers aren’t confined to a single codebase

38. Inspired by Joel Spolsky’s “The Joel Test: 12 Steps to Better Code”
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/
A practical checklist for
implementing inner source

39. ● Pick a single source manager for the entire organization
○ Preferably choose a stack-agnostic solution
○ Preferably choose a solution that supports managing and publishing documentation
○ Typically GitHub or GitLab (GitHub may be better if you plan on open sourcing later)
● Potentially a cultural barrier to cross
1. Does everyone in the organization have access to all of
the codebase by default?

40. ● Openness has little value without discoverability
● Choose a physical organization the team is familiar with
○ Don’t follow the organization chart, it will likely change over time
○ Ideally follow a reasonably stable functional layout (e.g., domain / product / component)
● Allows individual pieces to become “stepping stones” for others to build upon
2. Is the codebase structured or documented in a way that
makes it easy to explore?

41. ● Think of the single main problem your project is solving for the users
○ This is crucial to constrain the scope
○ NB: “continuing to exist” is not a mission
● Typically formulated in a README.md at the root of the repository
3. Do individual projects have a documented mission?

42. ● Low barrier of entry is key for a welcoming project
○ Anyone should be able to build, test, run with very few steps
○ Future hires to the team will thank you
● Typically formulated in a CONTRIBUTING.md at the root of the repository
4. Do individual projects have a documented and
straightforward contribution process?

43. ● If there’s two processes, you can be sure that one of them is obsolete
○ Spoiler: it’s the one used by people external to the project
● Applies to
○ Coding guidelines
○ Contributions expectation
○ SLA for patch review (recommandation: use FIFO is most cases)
○ Testing (are maintainers relying on a dedicated QA team?)
● Don’t blame external contributors for not knowing your undocumented rules
5. Are all contributors treated equally (i.e., teams /
maintainers follow a different process)?

44. ● Organise your company communications for transparency & discoverability
○ Encourage conversations to be public by default
○ Make the channels easy to discover (e.g., use a common format for teams)
● Document design and architecture decisions
○ Can be markdown files in git, or simply issue comments
6. Is there an open discussion platform where decisions are
taken and can be challenged?

45. ● Good rules
○ Are collectively defined and managed as code
○ Start small, and grow in scope as we learn from experience
○ Foster collaboration while leaving space for self-organization
■ For example: to enforce the previously mentioned items of the checklist
● Who is the “fair authority” inside your organization?
○ Ideally a group of highly respected technical leaders (preferrably IC over managers)
7. Is there a set of commonly agreed upon rules enforced
by a fair authority?

46. Inner source: a checklist
1. Does everyone in the organization have access to all of the codebase by default?
2. Is the codebase structured or documented in a way that makes it easy to explore?
3. Do individual projects have a documented mission?
4. Do individual projects have a documented and straightforward contribution process?
5. Are all contributors treated equally (i.e., teams / maintainers follow a different process)?
6. Is there an open discussion platform where decisions are taken and can be challenged?
7. Is there a set of commonly agreed upon rules enforced by a fair authority?

47. Final words
Ignoring wise crowds patterns Applying wise crowds patterns
Open source software
Pushes the community away
Will inevitably die
Scales to thousands contributors
Maintains quality and velocity
Has extreme resilience
Closed source software
Relies on tribal knowledge
Produces lower quality output
Tends to build castles over cities
Allows to scale wisely
Puts group’s interest over teams
Raises the bar for best practices

48. Final words
Open source communities have repeatedly shown a capacity to produce high
quality sustainable software by carefully designing for collective intelligence
These recipes apply to the enterprise, but it’s a matter of culture
● Requiring full support from leadership
● Immune to whatever tool you may throw at it

49. Arnaud Porterie - @icecrime
Thank you!