1. Updating Embedded Linux devices
in the field
Updating Embedded Linux devices in the field 1 Copyright © 2011-2015, 2net Ltd

2. License
These slides are available under a Creative Commons Attribution-ShareAlike 3.0
license. You can read the full text of the license here
http://creativecommons.org/licenses/by-sa/3.0/legalcode
You are free to
• copy, distribute, display, and perform the work
• make derivative works
• make commercial use of the work
Under the following conditions
• Attribution: you must give the original author credit
• Share Alike: if you alter, transform, or build upon this work, you may distribute
the resulting work only under a license identical to this one (i.e. include this
page exactly as it is)
• For any reuse or distribution, you must make clear to others the license terms of
this work
Updating Embedded Linux devices in the field 2 Copyright © 2011-2015, 2net Ltd

3. About Chris Simmonds
• Consultant and trainer
• Working with embedded Linux since 1999
• Android since 2009
• Speaker at many conferences and
workshops
"Looking after the Inner Penguin" blog at http://2net.co.uk/
https://uk.linkedin.com/in/chrisdsimmonds/
https://google.com/+chrissimmonds
Updating Embedded Linux devices in the field 3 Copyright © 2011-2015, 2net Ltd

4. Overview
• The need for software update of embedded devices
• Update by file or system image
• Update agent: bootloader or application
• An Linux update agent: SWupdate
• Update strategies
• Failsafe
Updating Embedded Linux devices in the field 4 Copyright © 2011-2015, 2net Ltd

5. The problem
• Embedded software is non-trivial (=> has bugs!)
• Devices are often connected to the Internet
• Allowing hackers to exploit the bugs
• Result: problems with the IoT
• We need an update mechanism to fix those bugs in
the field
• (Not to mention the desire to deploy new features,
better performance, etc.)
Updating Embedded Linux devices in the field 5 Copyright © 2011-2015, 2net Ltd

6. Components of embedded Linux
• Bootloader
• Linux Kernel (+ Device Tree and kernel modules)
• Root file system (basic operating system libraries and
utilities, usually read-only)
• System apps (often a separate partition, but may be
combined with root file system)
• User data (configuration settings, logs, user-supplied
data)
Updating Embedded Linux devices in the field 6 Copyright © 2011-2015, 2net Ltd

7. Frequency of update
Frequency
Ease of update
Bootloader
Kernel
Root file system
System applications
Updating Embedded Linux devices in the field 7 Copyright © 2011-2015, 2net Ltd

8. Requirements of an updater
• Robust: must not render the device unusable
• Atomic: must not apply half an update
• Fail-safe: have a fall-back mode if all else fails
• Remote, unattended update (as an option)
• Audit trail: so you can tell what updates have been
applied
• Preserve user data
Updating Embedded Linux devices in the field 8 Copyright © 2011-2015, 2net Ltd

9. A simple approach
• Majority of updates are to root filesystem and system
apps, which are "just files"
• Many projects have an updater based on tarballs and
some shell scripts
• They all fail, because:
• Lack atomicity
• Can’t update kernel in raw flash memory
• (Usually) no audit trail
• Not fail-safe
• Remember: if it can go wrong, it will go wrong
(eventually) 1
1Murphy’s law
Updating Embedded Linux devices in the field 9 Copyright © 2011-2015, 2net Ltd

10. Atomic file updates
• Update is either complete or not done at all
• "Easy" to achieve for individual files, e.g. using
POSIX rename (2) and careful checkpointing with
fsync (2)
• More difficult to keep relationships between files
consistent
• Example: application and dependant libraries
• (Can be done if you pay attention to library versioning)
• Mainstream Linux package managers (RPM and
dpkg) solve a lot of the problems but are still not
bulletproof
Updating Embedded Linux devices in the field 10 Copyright © 2011-2015, 2net Ltd

11. Atomic system image updates
• Instead of a large number of individual files, consider
a system as a smaller number of system images
• Kernel, root filesystem, system apps, etc
• Embedded devices are typically put together like this
• Need a method to update safely
• Examples:
• Dual copies of OS and ping-pong between them
• Have separate recovery OS just for updating
Updating Embedded Linux devices in the field 11 Copyright © 2011-2015, 2net Ltd

12. Two copies of the OS
Bootloader User
data
Boot
flag
OS Copy 1
OS Copy 2
• Update OS copy 2 from OS copy 1; set boot flag
when done
• "ping-pong" between OS copies
• But, requires double the flash storage
Updating Embedded Linux devices in the field 12 Copyright © 2011-2015, 2net Ltd

13. Recovery OS
Bootloader
Main OS
Recovery OS
User
data
Boot
flag
• To perform update, set boot flag and reboot into
recovery OS
• Then update main OS; clear boot flag when done
• Reboot to start updated main OS
Updating Embedded Linux devices in the field 13 Copyright © 2011-2015, 2net Ltd

14. Bootloader as update agent
• Seems an attractive option: simple environment with
no OS running
• But, bootloaders have limited support for peripherals,
file system formats and network protocols
• Incorporating the necessary software would make the
bootloader more buggy
• Remember that updating the bootloader itself is
difficult, often a single point of failure
• Conclusion: bootloader is not a good update agent!
Updating Embedded Linux devices in the field 14 Copyright © 2011-2015, 2net Ltd

15. Linux application as update agent
• Much better environment
• Access to all peripherals
• Read-write access to file systems
• All network protocols
• Maybe a display for user interaction
• Few open source examples
• One is SWupdate
Updating Embedded Linux devices in the field 15 Copyright © 2011-2015, 2net Ltd

16. SWupdate
• Written by Stefano Babic
• Source: https://github.com/sbabic/swupdate
• Doc: https://github.com/sbabic/swupdate/blob/
master/doc/source/swupdate.rst
• Yocto meta layer available:
https://github.com/sbabic/meta-swupdate
• Can use pre- and post- install scripts to set/reset the
boot flag in the bootloader (U-Boot, for example)
Updating Embedded Linux devices in the field 16 Copyright © 2011-2015, 2net Ltd

17. Local update
Main OS
image
Recovery OSSD
SWupdate
• Update supplied on local removable media, initiated
by user or maintenance staff
Updating Embedded Linux devices in the field 17 Copyright © 2011-2015, 2net Ltd

18. Remote update 1/2
Main OS
image
Recovery OS
SD
SWupdate
Downloader
Updater
• Download update from network to local storage:
allows update to be verified and installed at a
convenient time
• May be initiated by user or automatically
• Requires spare local storage
Updating Embedded Linux devices in the field 18 Copyright © 2011-2015, 2net Ltd

19. Remote update 2/2
Main OS
image
Recovery OS
SWupdate
Downloader
Updater
• Update directly from network stream: no local
storage needed
• Device is unavailable while updating, may lead to
significant down time if link slow or broken
• (Streaming not implemented in SWupadte yet)
Updating Embedded Linux devices in the field 19 Copyright © 2011-2015, 2net Ltd

20. Failsafe
• Robust software update is not the whole answer
• Can’t protect against installing (correctly) a buggy
update
• Last line of defence:
• Watchdog -> reboots if device is not functioning
• Bootloader counts boots triggered by watchdog
• Boot into recovery mode when exceed threshold and
attempt to re-install
Updating Embedded Linux devices in the field 20 Copyright © 2011-2015, 2net Ltd

21. Further reading
• "Building Murphy-compatible embedded Linux
systems" by Gilad Ben-Yossef https://www.kernel.
org/doc/ols/2005/ols2005v1-pages-21-36.pdf
• "Updating an embedded system :swupdate
documentation"
http://sbabic.github.io/swupdate/
• The rename (2) function:
http://pubs.opengroup.org/onlinepubs/
009695399/functions/rename.html
• The fsync (2) function: http://pubs.opengroup.org/
onlinepubs/009695399/functions/fsync.html
Updating Embedded Linux devices in the field 21 Copyright © 2011-2015, 2net Ltd

22. • Questions?
Slides on Slide Share: http://www.slideshare.net/
chrissimmonds/linux-field-update-2015
Updating Embedded Linux devices in the field 22 Copyright © 2011-2015, 2net Ltd