Another step towards immutability with Flatcar Container Linux

From the original CoreOS to the CNCF ecosystem

#

If you are familiar with Talos Linux, you will already be aware of the concepts associated with immutability and declarative state management, often described as-code. I’ve previously discussed these concepts in the context of setting up Kubernetes clusters; however, there’s one part of the infrastructure that’s often overlooked: so-called satellite services, those running outside Kubernetes, which also deserve the same level of attention. This is exactly where Flatcar Container Linux comes into the spotlight.

To understand Flatcar, you need to go back a few years. It all began with CoreOS Container Linux, a Linux distribution designed from the outset to run containers at scale.

CoreOS introduced some rather revolutionary concepts compared to traditional distributions: a read-only root filesystem, atomic updates via A/B partitioning, and declarative configuration at boot time.

Unfortunately, Red Hat acquired CoreOS in 2018 before announcing its end of life two years later, in order to prioritise its own Fedora-based distribution and in-house tools such as Podman. However, they hadn’t reckoned with the Berlin-based start-up Kinvolk, that forked the project to give it a new lease of life under the name Flatcar Container Linux. Microsoft subsequently acquired Kinvolk in 2021, before Flatcar joined the Cloud Native Computing Foundation (CNCF) in 2024 as an incubation project.

This CNCF status is significant. It guarantees the project’s neutrality with regard to commercial players, ensures open governance and offers long-term visibility comparable to that of other projects such as containerd, Argo CD, or even Cluster API itself.

Flatcar’s philosophy is directly inherited from CoreOS: to do away with pet servers, those machines that are looked after, patched over the months, and whose actual condition nobody really knows. Instead, Flatcar adopts the cattle model: disposable, interchangeable servers whose configuration is defined at start-up and which do not deviate from their initial state.

Flatcar Container Linux vs Talos Linux: two different approaches to immutability

#

If you’re reading this article after the Cluster API series with Talos Linux, one question naturally springs to mind: why use Flatcar if you already have Talos? The answer is quite simple and depends on the context of use.

Talos Linux is designed exclusively for Kubernetes. No SSH shell, no package manager, no direct access to the system: all management is carried out via talosctl and its gRPC API.

This is a radical approach that guarantees an extremely high level of security and immutability, but it means that Talos does only one thing: run Kubernetes nodes. However, version 1.15 is set to change the game with support extended beyond your favourite container orchestrator, though this is not yet the case currently.

Flatcar Container Linux takes a different approach. It is a general-purpose, immutable operating system for containers. The /usr filesystem is read-only, updates are atomic, but SSH access remains possible and a temporary debug container (toolbox) is available for one-off interventions. Flatcar supports Docker and containerd, manages its units via systemd and is designed for all containerised workloads. It is also supported, for example, by kubespray and Cluster API for creating Kubernetes clusters.

In practice, the use cases are quite distinct:

• Talos Linux is minimalist and designed for Kubernetes above all else. It is therefore the preferred choice for a minimal attack surface and a very high level of security.
• Flatcar can handle all container-based use cases with an immutable approach that is less restrictive than Talos. It has the advantage of being closer to traditional operating systems, while providing a well-designed automatic update system.

Here are a few very brief points to compare these two worlds, further details will follow below:

The two tools are complementary rather than competing. It is perfectly possible to have an infrastructure where Talos manages the Kubernetes clusters while Flatcar hosts the surrounding infrastructure services.

Now let’s move on to the key features…

Under the cover, the technical architecture

#

A/B partitioning and atomic updates

#

This is one of the features that sets Flatcar apart. The device is partitioned into two identical slots, USR-A and USR-B. At any given time, the system boots from one of them, leaving the other inactive. During an update, the new version is written to the inactive slot. The switch to this new slot only takes place at the next reboot.

What makes this mechanism particularly robust is its behaviour in the event of a failure. If the new system fails to boot correctly after a defined number of attempts, the bootloader automatically switches back to the old slot. There is no “half-updated” state, no corrupted configuration halfway through: either the system has booted correctly on the new version, or it has reverted to the old one.

The /usr partition is mounted as read-only. It is therefore impossible to install packages there or to edit files directly. This is a deliberate constraint that ensures the system’s state always matches what was defined during its provisioning.

OTA updates using Nebraska and the Omaha protocol

#

Flatcar’s Over-The-Air updates are based on the Omaha protocol, the same one used by Chromebooks and Google Chrome, and on an update server called Nebraska. The update_engine, the service in charge of updates, regularly queries Nebraska to check whether a new version is available.

Flatcar provides four distribution channels:

• Stable: the recommended channel for production, containing the most thoroughly tested versions;
• Beta: for validating new versions before promoting them to stable;
• Alpha: the very latest versions, for testing or development;
• LTS (Long-Term Support): for environments that prioritise long-term stability.

It is possible to deploy your own Nebraska server to precisely control when and how machines are updated, something that is essential in a production environment.

As for restarting when necessary, particularly in the case of a Kubernetes cluster, Flatcar Update Operator or the traditional Kured can be installed to manage and coordinate the restart of nodes.

In the case of standalone machines, it is possible to configure an automatic restart window within the update_engine configuration.

A reduced attack surface by design

#

Flatcar does not have a traditional package manager such as apt or yum. It is not possible to install additional tools directly on the host system unless specific directories such as /opt are used.

This restriction, while it may seem inconvenient at first, is in fact a security safeguard: an attacker who managed to gain access to the system would be unable to install additional tools via conventional methods.

For troubleshooting purposes

#

For situations where manual intervention is still required in the event of an error or the need for troubleshooting, Flatcar offers two mechanisms:

• The flatcar-reset command allows you to completely reset the system to its initial state, erasing the data on the user partition. This is equivalent to reprovisioning without having to recreate the machine.

• The other option is toolbox, accessed via the command of the same name, providing a temporary container that gives access to a complete environment with all the usual diagnostic tools (tcpdump, strace, etc.), without these tools necessarily being installed on the host system. In fact, this command runs a script that creates a container with privileged access and mounts the file system directly inside it.

OS extensions

#

The read-only nature of /usr raises an interesting question: how can additional tools or components be installed without breaking the immutable model?

Flatcar’s solution involves systemd-sysext, a native systemd mechanism that allows file system images to be overlaid onto /usr at boot time. These images, known as system extensions or sysext, are SquashFS files (.raw) that are activated as overlays on top of the standard system partition. These binaries and configurations appear to be installed directly within the OS, while the base image remains unchanged.

Flatcar offers the sysext-bakery project, which brings together recipes and ready-to-use images for common tools such as Keepalived or Kubernetes. The extensions are placed in /etc/extensions/ or /var/lib/extensions/, and the ensure-sysext.service service handles their activation at boot.

Furthermore, it is possible to take this a step further with systemd-sysupdate to manage extension updates autonomously, independently of the OS’s own update cycle.

Declarative configuration with Ignition

#

Flatcar uses Ignition for its initial configuration. More details are provided in the following section, but the idea is simple: the entire system configuration (users, SSH keys, files, systemd units) is defined before the first boot and applied once, atomically, when the virtual machine starts up.

Provisioning

#

What is Ignition?

#

Ignition is the initial configuration system for Flatcar Container Linux. Its key feature is the timing of its execution: It runs within the initramfs, handling the preparation and mounting of the disks even before switching to the root filesystem. This means that by the time systemd starts up for the first time, the system is already fully configured and ready to use.

This approach differs significantly from cloud-init, where the configuration runs after the system has booted, with the possibility of leaving the machine in a partially configured state in the event of an error.

With Ignition, it’s all or nothing: if the configuration is invalid or if a declared resource is inaccessible, the machine simply won’t boot. There is no intermediate state, no half-applied configuration. This deterministic behaviour greatly simplifies debugging and ensures consistency of state.

Ignition vs cloud-init

#

Here is a brief summary to help you understand the differences between these two modes of operation:

From Butane to Ignition

#

Nobody writes JSON without making mistakes… So writing Ignition JSON by hand would be nothing short of a nightmare!

However, to get a more readable format, you can use Butane, a tool that compiles YAML into valid Ignition JSON without any missing curly brackets!

Here’s a slightly more illustrative example, adding an SSH key to the default core user and changing the virtual machine’s hostname:

variant: flatcar
version: 1.1.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAA...

storage:
  files:
    - path: /etc/hostname
      contents:
        inline: my-server
      mode: 0644

You can convert everything using this command:

butane --pretty --strict config.bu > config.ign

The resulting Ignition JSON is then passed to the machine via its provisioning mechanism (userdata variable, HTTP URL, etc.). It is read only once on first start-up and is never reapplied, as mentioned above:

{
  "ignition": {
    "version": "3.4.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-ed25519 AAAA..."
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "path": "/etc/hostname",
        "contents": {
          "compression": "",
          "source": "data:,my-server"
        },
        "mode": 420
      }
    ]
  }
}

It’s easier this way, isn’t it?

A few words to conclude

#

Flatcar Container Linux fits perfectly with the philosophy of modern infrastructure: machines whose state is fully defined at source, that must not deviate from their initial configuration, and whose update cycle is atomic and reversible. No more overnight patching, no more manual configuration that builds up over the months, and no more wondering what is actually running on a machine.

For those wishing to explore the CNCF ecosystem further, Flatcar also offers providers for Cluster API using the Ignition format. Furthermore, it is entirely possible to provision immutable Kubernetes nodes with Flatcar as the base operating system, using the kubeadm bootstrap provider.

This is an interesting alternative to Talos Linux for teams looking for an immutable OS with SSH access and broader compatibility with the traditional Kubernetes ecosystem.

In a future article, I’ll show you how to deploy a Flatcar Container Linux instance as-code, what do you think?