My talk from the Silicon Valley LUG (and others) Meetup on March 9 2020.

1. Drew Moseley
Technical Solutions Architect
Mender.io
IoT: Contrasting Yocto/Buildroot to Binary OSes

2. Session overview
● Motivation
● Challenges for Embedded,
Linux and IoT developers
● Describe and compare IoT
system build workflows
● Do containers matter for IoT?

3. Thank you to our organizer!
Kevin Dankwardt and the following meetup groups:
● Silicon Valley Linux Technology
● SFBayLUG
● Silicon Valley IoT

4. About me
Drew Moseley
○ 10 years in Embedded Linux/Yocto development.
○ Longer than that in general Embedded Software.
○ Project Lead and Solutions Architect.
drew.moseley@mender.io
https://twitter.com/drewmoseley
https://www.linkedin.com/in/drewmoseley/
https://twitter.com/mender_io
Mender.io
○ Over-the-air update manager for
embedded Linux
○ Open source (Apache License, v2)
○ Dual A/B rootfs layout (client)
○ Remote deployment management (server)
○ Under active development

5. Embedded Projects increasingly use Linux:
● AspenCore/Linux.com1
: Embedded Linux top 2 in current and planned use.
Huge IoT market opportunity:
● Forbes2
: $267B by 2020
Linux is a big player in IoT
● Nodes & Gateways3
- 17.18 Billion units by 2023
● Inexpensive prototyping hardware - Raspberry Pi, Beaglebone, etc
● Readily available production hardware - Toradex, Variscite, Boundary
Devices
● Wide selection of chipsets - NXP, TI, Microchip, Nvidia
1
https://www.linux.com/news/event/elce/2017/linux-and-open-source-move-embedded-says-survey
2
https://www.forbes.com/sites/louiscolumbus/2017/01/29/internet-of-things-market-to-reach-267b-by-2020
3
http://www.marketsandmarkets.com/PressReleases/iot-gateway.asp
Motivation

6. Challenges for Embedded Linux/IoT Developers
Hardware variety
Storage Media
Software may be maintained
in forks
Cross development
Initial device provisioning

7. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
1
https://makezine.com/comparison/boards/

8. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
2. Connect Hardware
1
https://makezine.com/comparison/boards/

9. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
2. Connect Hardware
3. Install OS
1
https://makezine.com/comparison/boards/
a. Binary distribution
b. Build system

10. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
2. Connect Hardware
3. Install OS
a. Binary distribution
b. Build system
4. Develop and Test
1
https://makezine.com/comparison/boards/

11. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
2. Connect Hardware
3. Install OS
a. Binary distribution
b. Build system
4. Develop and Test
5. Deploy
1
https://makezine.com/comparison/boards/

12. Getting Started Guide for Embedded/IoT Development
1. Buy Hardware1
2. Connect Hardware
3. Install OS
a. Binary distribution
b. Build system
4. Develop and Test
5. Deploy
1
https://makezine.com/comparison/boards/
6. PROFIT!!!

13. Why are we here?
To build things.
Useful things.
But how?

14. Why are we here?
To build things.
Useful things.
But how?
It depends

15. System Requirements
Common:
● Off the shelf hardware
● Linux
● Limited functionality “appliance”
Unique:
● Fleet size
● Product lifecycle
○ POC vs Production
○ Legacy devices?
○ Lifetime
● Use cases

16. Option #1 - Binary Distros
● Installer or pre-built image from
board vendor.
● Boot board:
○ Remove things
○ Add things
● Create “Golden Master”
● Optional
○ Use Debian build tooling

17. Option #1 - Binary Distros - cont
Advantages:
● Simple
● Developer familiarity
● Quick
Disadvantages:
● Golden master bottleneck
● Reliance on external providers
○ System architecture
○ Availability of binaries
● Reproducibility
● Larger images
● License compliance

18. Option #2 - Build Systems
● Checkout build system
metadata.
● Configure
● Run “make” or equivalent
● Deploy and test
● Check in local changes
Linux system as code

19. Option #2 - Build Systems
Advantages:
● Configurability
● Reproducible
● No bottleneck on golden master
● No reliance on external providers
● Smaller images
Disadvantages:
● Complexity
● Learning curve
● Long build times
● Large build systems
● Unfamiliar working model (ie cross
compiling everything)

20. Build System Defined
_Is_
● Mechanism to specify and build
○ Define hardware/BSP
components
○ Integrate user-space
applications; including custom
code
● A system that:
○ Is reproducible
○ Supports multiple developers
○ Allow for parallel processing
● Ideally provides:
○ (Cross) Toolchains
○ License Management
_Is Not_
● An IDE
● A Distribution
● A deployment and provisioning
tool
● An out-of-the-box solution

21. “It’s not an embedded Linux distribution -- it creates a custom one for you”1
● Recipes, metadata, dependencies and configuration
● Primary output: package feed
● Secondary output: boot images
● Builds all components from source
● Mechanism, not policy
Products:
● Root filesystem image
● Kernel, Bootloader, Toolchain
● Package Feed
Yocto Project - Overview
1
See more at https://www.yoctoproject.org and https://openembedded.org

22. “Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux
systems through cross-compilation.”1
● Primary output: boot images
● Does not support rpm-style package mgmt
● “Firmware Generator”
● Builds all components from source
● Focus on simplicity
Products:
● Root filesystem image
● Kernel, Bootloader, Toolchain
Buildroot - Overview
1
See more at https://buildroot.org/

23. ● “The ‘s’ in IOT stands for security” - @tkadlec
● 1-25 bugs per 1000 lines of code*
○ Assume that all software components have
vulnerabilities
● Use well-maintained software and keep it updated
● Review vendors for update policies
● General Security Practices
○ Principle of least privilege
○ Separation of privilege
○ Kerckhoff’s principle
■ “You can only design an encryption system
that someone dumber than you cannot
crack.”
*Source: Steve McConnell, Code Complete
Security

24. ● “The ‘s’ in IOT stands for security” - @tkadlec
● 1-25 bugs per 1000 lines of code*
○ Assume that all software components have
vulnerabilities
● Use well-maintained software and keep it updated
● Review vendors for update policies
● General Security Practices
○ Principle of least privilege
○ Separation of privilege
○ Kerckhoff’s principle
■ “You can only design an encryption system
that someone dumber than you cannot
crack.”
*Source: Steve McConnell, Code Complete
OTA updates are
a must have.
Security

25. “A container is a standard unit of
software that packages up code and all
its dependencies so the application runs
quickly and reliably from one computing
environment to another.”1
Containers
1
https://www.docker.com/resources/what-container
“An operating system paradigm in which
the kernel allows the existence of
multiple isolated user space instances.”2
2
https://en.wikipedia.org/wiki/OS-level_virtualization

26. ● Isolation
○ File system
○ I/O
○ CPU
○ Network
● Dependency packaging
● Shared kernel
● Optional:
○ Orchestration
○ Network distribution
○ Global repositories
○ Private repositories
Containers - characteristics

27. ● chroot
● lxc/lxd
● Docker
● runc
● rkt
● systemd-nspawn
● ...
Containers - implementations
https://en.wikipedia.org/wiki/OS-level_virtualization lists 18 options

28. Server side:
● Definitely useful here.
● Same basic use cases as for any other use cases.
Containers in Embedded/IoT?
Client side:
● Limited utility due to limited functionality of devices.
● Custom distro should mean fewer issues due to clashing dependencies.
● Similar concerns as with binary distros.
⎻ Reproducibility
⎻ Availability
● Use cases:
⎻ Packaging of application code for updatability
⎻ Architecture separation
⎼ Common base system software
⎼ Device personality provided by containerized code

29. ● Define your application first
○ Please consider OTA updates early
● Get and use a reproducible build
system
● Keep an eye on containers
● Don’t Panic
● Call us
Conclusions

30. ● https://bit.ly/2GlKlUQ - Previous ELC Talk comparing Embedded Linux build
systems
● https://ubm.io/2Iazdfn - Deeper dive into the Yocto project
● https://hub.mender.io/t/raspberry-pi-3-model-b-b/57 - Building Yocto for
Raspberry Pi with Mender.
● https://www.linuxjournal.com/content/linux-iot-development-adjusting-bin
ary-os-yocto-project-workflow - Article discussing the two workflows
For more information:

31. Questions?
Thank you!
@drewmoseley
https://mender.io
drew.moseley@mender.io