The Year of the Linux Desktop may get postponed every year, but Linux has come to dominate pretty much every other market segment - from servers, industrial machines and mobile phones (via Android), to a host of embedded devices.
In the modern connected world, it is more important than ever to make sure all devices are kept up-to-date with all the latest security and feature updates. In this article, we’ll dive into the possibilities of managing system updates on various flavors of Linux.
But first, let’s see how it’s different compared to smaller embedded devices.
Broadly speaking, there are two main types of architecture that can be used for embedded (IoT) devices:
Of course, you can also run an RTOS on a large SoC or even a desktop computer, and there are variants of Linux which fit on some MCUs.
For those from the MCU/RTOS world, the “firmware” will be that single executable that you build when linking your RTOS, drivers, libraries and application code. It usually lives in a specific range of flash memory addresses, with the rest often reserved for the bootloader, swap space for firmware updates, and non-volatile settings storage. There are a couple of possible approaches to firmware updates, but the general idea is to replace one binary with a different version of it. Those binaries are just blobs of data, and because there is a single binary being built, versioning is pretty simple - when you build firmware, you give it a version number and that’s it. You can’t upgrade the RTOS kernel without rebuilding the entire firmware, so all dependency management and version control has to happen at compile time, on developer machines.
Users of desktop computers will know that larger OSes are vastly different. In fact, when talking about full-fledged computers, the operating system is rarely referred to as “firmware” at all. The term tends to be reserved for BIOS or UEFI software that lives on your motherboard and is responsible for early initialization of the hardware. But it is not very useful as an operating environment in and of itself.
So let’s see how all this maps onto various types of Linux systems. To get a baseline, we’ll start with the traditional desktop and server distributions.
A typical desktop Linux distribution - think Ubuntu, Fedora, Arch, or even Raspberry Pi OS - is, in essence, a collection of packages. You may be familiar with *.deb files and the apt tool on Ubuntu or Debian, or *.rpm and dnf on Fedora. apt and dnf are examples of package managers, which install, uninstall and update packages while also ensuring that they play nicely together - for example, by enforcing dependencies.
Packages exist for common software like the Firefox browser or LibreOffice, as well as core system components like the Linux kernel, the C library or the GRUB bootloader. Those core component packages will often include additional scripts responsible for updating the boot partition or reconfiguring the bootloader to use a different kernel version by default. However, the main point remains that the installed system is a collection of packages, each versioned separately. Many packages will declare dependencies on each other, e.g. a web browser may require the C library in a version of at least 2.17, but earlier than 3.0. The job of a package manager is to ensure that they aren’t broken.
Security and feature updates will be delivered by pushing newer package versions into a central repository. Distribution maintainers will make sure that all packages currently in the repository work with each other, so that users shouldn’t have dependency issues. Some distributions, like Arch, are considered “rolling release” - packages are released as software is updated upstream, so both minor security updates and major feature upgrades are delivered the same way.
However, in more traditional distributions like Ubuntu, there will be separate repositories for each of the OS releases - most mainstream distributions are released twice a year. Core packages are usually not updated to new major versions within an OS release. However, an in-place upgrade to a newer release is relatively similar - the OS is reconfigured to use the repositories of a newer release and an otherwise mostly normal system upgrade is performed - possibly with some additional scripting to take care of any major architectural changes, like when various distributions have changed the init system from sysvinit to Upstart and then to systemd.
This approach is very flexible, as it allows distribution maintainers to push updates to each system component separately, keeping the update downloads small. However, it also means that each individual installation is unique, and update issues may not be easily reproducible. The user is also generally expected to be present during the update process, to address any issues that may arise. For example, if the configuration files have been modified in a way that the package manager can’t handle, making it unsuitable for automated remote updates. Not only that, but there is very little resilience to errors - an unsuccessful update is likely to leave the system in an inconsistent and possibly unbootable state. Traditional distributions also usually don’t enforce the system integrity in any way - you may install things that aren’t packages (e.g. compile software yourself), for which the dependencies won’t be tracked in any way - subsequent system upgrades may easily break that software or even cause conflicts.
Modern distributions will also support containerized application packages like Snap or Flatpak, which avoid using the distribution’s native core packages as much as possible. Instead, they run applications in their own sandboxes with dedicated core components - different versions of which can be run side by side. I’ll leave these issues aside for the time being, though.
Commercial operating systems have long alleviated these problems by introducing mechanisms that prevent key system files from being modified. You might have heard about the System File Checker on Windows or System Integrity Protection on macOS. If you try to mess with files in the C:\Windows\System32 or /System directories, the OS will prevent you from causing damage, even if you’re running with administrative privileges.
Not only that - on macOS Catalina and later, the base system installation lies on a separate, read-only volume, and any modifications made by the user are overlaid on top of it but logically separated. Windows does not have such a feature on the file system level - but it keeps track of what a “clean” system is supposed to look like, so you can use the “Reset This PC” feature to functionally restore it, even without any installation media or downloads.
Since the mid-2010s, various Linux distributions have attempted to replicate those principles – often called “immutable” or “atomic” distributions – in the world of free software. The basic idea is to have most of the file system read-only, and encourage the user to install software in a containerized manner. However, the details depend heavily on a specific distribution. Here are a couple of examples:
All of these approaches have various tradeoffs between flexibility, reliability, and storage efficiency. However, each is designed to be directly accessed and maintained by the end users. In the world of embedded systems, reliability and predictability often take priority, leading to automated, remote updates which are executed without the user interacting with the system directly. On the other hand, flexibility is usually not as important, as the software is generally not installed or uninstalled as part of normal usage.
There is no single handbook for every application. Some embedded systems may opt for traditional package-based distributions if a particular application requires such flexibility. Others may opt for filesystem-level immutability, using solutions such as OSTree.
However, the most common update paradigm seems to be block-level updates of immutable partitions. This comes in two main variants:
Note that the partition layout may be more complicated than just two “A/B” or “main+recovery” partitions. Actual layouts usually also include a partition for non-volatile data such as configuration files or user storage, and an additional partition for the bootloader. Separate from the main system image may also be found the Linux kernel, peripheral firmware and additional applications. In the case of the A/B paradigm, those may be either duplicated alongside each of the A/B root filesystems or shared between both.
The root filesystem partition itself may or may not be mounted read-only on a running system. Sometimes it’s easier to mount it read-write to allow software that e.g. writes system logs and other volatile data onto disk to run unmodified. However, in such a case it is expected that all such data will be lost after an upgrade, so actually non-volatile data need to reside on a separate read-write partition, and the software directed to use it - either via compile-time configurations, symlinks on the root partition, or mounted as an overlay.
As already mentioned, the A/B paradigm is also the basis of system upgrades on ChromeOS, Android and SteamOS, among others. On the other hand, the “main+recovery” paradigm should be familiar to laptop users - a recovery partition that allows the OS to be reinstalled from the Internet has been standard on Apple computers since Mac OS X Lion (2011). Windows Recovery Environment, though somewhat more limited, has been available since Windows Vista (2006), while Windows laptop vendors were known to introduce custom recovery partitions years prior to that.
The A/B paradigm should also feel familiar to those experienced in solutions based on MCUs and RTOSes. However, there are some differences in the upgrade flow. On RTOS-based devices, the bulk of the upgrade logic is contained in the bootloader such as MCUboot or X-CUBE-SBSFU. The typical flow involves writing the raw update image onto the inactive partition, and the bootloader handles the rest after a reboot which may involve swapping the two partitions around, verifying checksums and cryptographic signatures, or even decrypting an encrypted image. On Linux-based devices, all these tasks are typically performed by the updater software before the reboot.
The simplest form of such “updater software” is just performing a bit-for-bit dd copy from the update image onto the system partition, and perhaps switching a flag in the bootloader to boot from the previously inactive partition. However, there are a number of additional features that are desirable for a production solution, including:
Additionally, where tamper-resistance is important, the Linux kernel provides an additional layer of security - the dm-verity module. This allows the creation of a hash tree, which allows detection of any data modification on the protected block device.
dm-verity is enabled by default on Android and ChromeOS devices, and can also be used on any embedded device. When it is combined with other hardening techniques such as secure boot and kernel module signing, a device that will only ever run verified and signed file system images can be created.
To support all those features, a specialized class of software for managing software updates on embedded Linux systems emerged.
In this section you will learn about some of the readily available open-source software for managing updates on embedded Linux systems.
According to its Github page, SWUpdate is “a Linux Update agent with the goal to provide an efficient and safe way to update an embedded Linux system in the field”. The project itself is licensed under GPLv2, but can also be controlled via a library that is released under LGPLv2.1. The first release dates back to July 2014, and the project has been being actively developed ever since, with a new stable version typically released twice a year.
It is designed for block-level upgrades for immutable filesystems. Its explicit design goal is that “SWUpdate should be configured to fit into a project, not vice versa”, so it can be made to run on pretty much any Linux system, in many paradigms including both “A/B” and “main+recovery”. The SWUpdate project itself does not impose any design choices upon the firmware vendor, so you still need to decide, for example, how to handle non-volatile storage.
Most integrations are based on either Yocto or, less often, Buildroot. The swupdate package is available in upstream Buildroot repository, but all configuration of both the system (partitioning etc.) and the updater has to be provided manually. Likewise, the basic integration with Yocto is available as a meta-swupdate layer; simple configurations for Raspberry Pi and Beaglebone devices are available as meta-swupdate-boards, but these are only intended as examples, not as production-ready configurations.
SWUpdate is modular to an extent, and contains the following parts:
SWUpdate defines a uniform update file format, which typically uses the *.swu extension. Under the hood, it is a cpio archive that may contain any number of archives, as well as a configuration file that describes how they should be applied. Depending on the SWUpdate’s compile-time configuration, the configuration file may support arbitrary scripting in Lua or Unix shell, as well as signing and/or encrypting of the packages, in which case proper signature or encryption will be required for successful upgrade. This file format is designed to be “streamed” linearly – that is, applied on the fly as it is being downloaded, without the need for any intermediate storage.
SWUpdate also supports “delta” images in two forms - the images can be rdiff images, utilizing the same algorithm as rsync, to be applied over a known previous state. Alternatively, the *.swu file may only contain the metadata while the actual data is synchronized over the network using a zchunk format.
SWUpdate is widely used in embedded Linux ecosystems, especially those based on Yocto.
RAUC (Robust Auto-Update Controller) is another solution that, in the words of its maintainers, “controls the update process on embedded Linux systems”. It is a newer project, with the first release dated February 2017, and is updated multiple times a year. It is licensed as LGPLv2.1.
RAUC is developed by Pengutronix and integrated with their own embedded Linux build system, PTXdist. Official integrations are also readily available for Yocto (meta-rauc), Buildroot, Debian, Ubuntu, Arch Linux and NixOS. The Yocto integration is additionally served by example configurations for a number of popular development platforms, provided through the meta-rauc-community repository.
Much like SWUpdate, RAUC supports both “A/B” and “main+recovery” paradigms and multiple partitions. Its update file format, *.raucb (RAUC bundle) is a SquashFS image that can additionally contain its signature, verity hash tree, and may also be encrypted. Within that bundle, there is a manifest file that describes other components, and any number of images in a variety of formats - either bit-for-bit filesystem images or directory archives.
Update bundles can be local files, or downloaded on-demand from an HTTP(S) server. However, unlike SWUpdate, on-demand downloads are handled through an integrated network block device driver and HTTP Range requests. RAUC relies on the ability to mount the SquashFS bundle, so linear streaming is not possible without caching the entire bundle locally.
Delta upgrades can be handled through either a built-in “adaptive updates” mechanism, or through casync or desync. In all three cases, download optimization is performed locally and the update bundle needs to be available in its entirety on the update server.
RAUC can be controlled either from the command line or via D-Bus. It is only concerned with applying the updates, so there is no integrated web server or automatic update checks. In general, it is less flexible, but also more lightweight than SWUpdate.
RAUC is used to update the Steam Deck and Deutsche Bahn ICE train information terminals.
Mender comes up often in the context of OTA updates on embedded Linux devices. Developed by Northern.tech, it is both an open-source project and a commercial product, designed with simplicity in mind. While SWUpdate or RAUC provide extensive configuration options to fit any possible project, Mender focuses on the typical use case of updating systems based on either Yocto or Debian, using OTA firmware and software management through Mender’s own server software.
Both Mender client and server are open source, licensed under Apache License 2.0. However, Northern.tech heavily promotes its own hosted instances of the Mender server, which offer additional commercial features. The client software is also tightly coupled with the server tech, although the client program also supports performing standalone updates from local files or arbitrary URLs.
Mender artifacts are signed but do not support encryption. They can contain full partition images (designed for the A/B paradigm), delta images, or “Update Modules”, which support arbitrary scripting and may be used for application-level updates. Full partition updates are streamed over the network, so the entire file need not be saved in intermediate storage.
Mender may seem like a very compelling option thanks to its simplicity and low barrier of entry. However, the tight coupling between the client and server sides, combined with not being based on open standards, means vendor lock-in may be an issue. It is also much less flexible and configurable than either SWUpdate or RAUC.
At AVSystem, we believe that OMA SpecWorks’ LightweightM2M is the right solution not only for remote management and monitoring of IoT devices, but also for managing the OTA firmware update process. Our Coiote IoT Device Management Platform offers a comprehensive solution for managing firmware update campaigns on both single devices and entire fleets.
Our LwM2M client library Anjay has been most popular on systems based on MCUs and RTOSes, but it is highly portable and can work on embedded Linux systems as well. Anjay implements a Firmware Update state machine compliant with the open standard OMA LwM2M protocol, which allows the user to implement the callbacks to any backend they want. This may very well be SWUpdate, RAUC, Mender’s standalone updater, or any other update solution for embedded Linux.
As you can read in the tutorial, Anjay handles the download itself and, through the aforementioned callbacks, passes the firmware image as a linear stream of bytes to the application code. For this reason, this implementation will work best with update agent software that enables linear streams to be applied on the fly, like SWUpdate or Mender.
The table below shows a comparison of how the callbacks can be implemented for different target platforms:
There are additional possibilities for managing updates in the LwM2M ecosystem. Multi-component Firmware Update has recently been introduced in Anjay and Coiote, which may be useful for updating non-monolithic systems, for example when some applications have a different release cycle to the OS itself. On the other hand, the LwM2M Software Management specification may be well suited for systems based on more traditional packages (*.deb, *.rpm, etc.) or containers (e.g. Docker).
The open-source nature of Linux means that it is endlessly adaptable and can fit almost any possible application. It also means that there are a multitude of possible strategies for updating the system. It’s important to understand the advantages and weaknesses of different solutions, and choose one that is appropriate for your particular application. Open-source firmware management software and readily available OTA update server solutions can help bring that solution to life, while staying true to the best practices used in the embedded Linux ecosystem.
Subscribe to stay in the loop with all our latest content: