↓ Skip to main content
Filador

First deployment with Flatcar Container Linux

·14 mins
kubernetes flatcar container cncf linux talos infrastructure immutable security
Romain Boulanger
Author
Romain Boulanger
Infra/Cloud Architect with DevSecOps mindset
Table of Contents
Flatcar Container Linux - This article is part of a series.
Part 1: Another step towards immutability with Flatcar Container Linux
Part 2: This Article

Introduction: from theory to practice
#

In the first article on Flatcar Container Linux, the theoretical foundations of this immutable operating system were laid out: a read-only /usr directory, an A/B partitioning scheme enabling atomic updates with automatic failover in the event of failure, and deterministic provisioning handled by Ignition right from the initramfs.

Theory is all well and good, but practice is even better! The aim of this second part is to deploy an initial Flatcar Container Linux virtual machine on Proxmox VE in a fully declarative manner using OpenTofu.

Rather than writing a Butane file by hand, there is the community provider community-terraform-providers/ignition, combined with the bpg/proxmox provider, to deploy a virtual machine. This allows you to generate the Ignition configuration directly in HCL, making use of variables and templating functions, all within my favourite infrastructure-as-code language!

To make this exercise as practical as possible, I propose deploying the Blocky DNS service, shipped with great features that include blocking adverts and unwanted domains using hosts files that you define. This service is based on a container and a YAML file for configuration, making it an ideal candidate for this scenario.

Prerequisites and architecture
#

Before writing the first lines of code, here is what you will need to carry out this deployment:

  • A working Proxmox VE node (8.4 or later) – that goes without saying;
  • The tofu binary installed (version 1.10 or later);
  • An SSH key pair loaded into your local SSH agent, specifically to connect to the Proxmox hypervisor and inject the Ignition configuration into the Datastore;
  • A free IP address on your local network to be assigned to the DNS server;
  • A Proxmox user and API token with Sys.Audit and Sys.Modify permissions on the / path; these privileges are required by the endpoint used to download the Flatcar image;
  • The import content type enabled on the datastore that will receive the Flatcar image.

The last two points are the easiest to overlook and will result in HTTP 403 and 500 errors on the first tofu apply command. The corresponding pveum and pvesm commands are detailed in the README.md file of the repository presented below.

One important point to note before going any further: from the Flatcar project’s perspective, Proxmox VE is one of the platforms supported by the community. This means that the dedicated image is indeed produced with every release, but it does not undergo the Flatcar team’s official suite of automated tests.

In practice, this works very well, but it’s best to be aware of this before rushing headlong into production.

Project structure
#

All the code covered in this article is available in this GitHub repository:

GitHub Repository Thumbnail
axinorm/flatcar-tofu

Deploy your first Flatcar Container Linux virtual machine with OpenTofu!

HCL
0
0

To ensure a clear and maintainable structure, responsibilities are split across several files:

├── README.md
├── backend.tf
├── encryption.tf
├── files/
│   └── blocky.yml
├── ignition.tf
├── image.tf
├── main.tf
├── outputs.tf
├── providers.tf
├── terraform.tfvars.example
└── variables.tf

The state encryption configuration can be found in encryption.tf, the backend declaration in backend.tf, the version and provider declarations in providers.tf, the download of the Flatcar image in image.tf, all the configuration generation in ignition.tf, and finally the instantiation of Ignition via a snippet and of the virtual machine in main.tf.

All default values can be found in terraform.tfvars.example. Of course, you’re free to adjust these values!

Configure the providers and add a layer of security
#

Let’s get straight to the point with the providers.tf file, where the version of the ignition provider is deliberately pinned at ~> 2.6.0 rather than the latest 2.7.x branch. I’ll come back to this in the section on Ignition, but it relates to the version of the JSON specification generated by the provider.

The ignition provider requires no additional parameters, unlike the proxmox provider, where the proxmox_api_token variable must be passed as an environment variable via a .envrc file to make life easier:

export TF_VAR_proxmox_api_token="tofu@pve!provider=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

In addition to this, the encryption.tf file adds encryption not only to the state but also to the plan.

In fact, when using Ignition with OpenTofu, the entire configuration rendered as JSON is stored in plain text in the state file or tfstate. Even though the only thing injected here is an SSH public key, it is very common to include passwords, TLS certificates or even tokens…

With OpenTofu, and this is why I much prefer it to Terraform, this feature is included by default.

Finally, even when using a local backend for this type of project in Lab mode, the .tfstate file is fully encrypted and can even be stored in a private Git repository without any risk of secrets being leaked. In a production environment, however, I would still recommend combining this encryption with a remote backend, particularly for concurrency concerns.

The Flatcar image
#

Now that the providers are ready, let’s retrieve the Flatcar Container Linux image designed for Proxmox VE. There’s no need to connect to the hypervisor via SSH to run a wget command manually: the proxmox_download_file resource in image.tf takes care of this directly:

locals {
  flatcar_image_url = "https://${var.flatcar_channel}.release.flatcar-linux.net/amd64-usr/${var.flatcar_version}/flatcar_production_proxmoxve_image.img"
}

resource "proxmox_download_file" "flatcar" {
  [...]
  content_type = "import"
  url          = local.flatcar_image_url
  file_name    = "flatcar_production_proxmoxve_image-${var.flatcar_channel}.qcow2"

  overwrite           = false
  [...]
}

This image can be selected using, amongst other things, the variables flatcar_channel (which accepts stable, beta, alpha or lts) and flatcar_version.

Two important points:

  • Firstly, the file_name attribute: the image published by Flatcar is called flatcar_production_proxmoxve_image.img, although it is actually an image in QCOW2 format. However, Proxmox VE relies strictly on the file extension to determine the import format and will simply reject a .img file. This means you need to rename it to .qcow2;

  • Next, overwrite = false: as the URL points by default to the current version, the remote file changes every time Flatcar publishes a new version. Leaving this attribute set to true would trigger a new download as soon as a new version becomes available.

This is where the link to the first part becomes meaningful: this base image is only used to instantiate the machine on the very first boot. Once the virtual machine has started, it is Flatcar’s internal update_engine service that handles updates to the passive partition.

You therefore do not need to download every new image or redeploy your machines to keep up with OS security patches.

Ignition and its configuration
#

Let’s move on to the core of the deployment with the ignition.tf file. The community-terraform-providers/ignition provider has a distinctive feature: it has no resources, only data sources.

It is a purely logical provider that assembles data structures and returns JSON compliant with the Ignition specification, without any comma errors or unclosed braces…

A quick note on versions: version 2.7.x of the provider generates JSON stamped with the Ignition 3.6.0 specification. However, the current stable branch of Flatcar (version 4757.2.2 at the time of writing) includes Ignition 2.24.0, which supports the specification up to version 3.5.0.

If you use version 2.7.x of the provider, Ignition will reject the configuration in the initramfs and the machine will not boot any further.

You can therefore see why the provider has the rule ~> 2.6.0 rather than the latest version.

Here is a summary of the data sources used in the ignition.tf file:

Data sourceRole in the configuration
ignition_userConfigures the core system user and associates the SSH public key with it
ignition_directoryCreates the /etc/blocky directory on the root filesystem
ignition_fileWrites the configuration files (/etc/hostname, network, resolved, Blocky, update.conf)
ignition_linkCreates the symbolic link /etc/localtime to set the time zone
ignition_systemd_unitDeclares and activates the systemd unit blocky.service
ignition_configMerges all of the above elements into the final JSON document

To begin with, take a look at the main sections of the ignition.tf file, starting with the default user core and the machine name:

data "ignition_user" "core" {
  name                = "core"
  ssh_authorized_keys = [trimspace(file(pathexpand(var.ssh_public_key_path)))]
}

data "ignition_file" "hostname" {
  path = "/etc/hostname"
  mode = 420 # 0644

  contents {
    source = "data:;base64,${base64encode("${var.vm_name}\n")}"
  }
}
[...]

File permissions (mode) are expressed in decimal notation in the provider: 420 corresponds to 0644 in octal (0o644), and 493 corresponds to 0755 (0o755).

The network is then configured via a file called /etc/systemd/network/00-static.network:

data "ignition_file" "network" {
  path = "/etc/systemd/network/00-static.network"
  mode = 420

  contents {
    source = "data:;base64,${base64encode(<<-EOT
      [Match]
      Name=en*

      [Network]
      Address=${var.vm_ip_address}
      Gateway=${var.vm_gateway}
      DNS=${var.host_dns_server}
    EOT
    )}"
  }
}

An additional file is required. This is because Flatcar starts systemd-resolved with its stub listener, which is already listening on 127.0.0.53:53. However, Docker will attempt to publish port 53 on 0.0.0.0, covering this address: the bind fails and the Blocky container will never start. This means the stub listener must be disabled using a specific configuration such as this:

data "ignition_file" "resolved_stub" {
  path = "/etc/systemd/resolved.conf.d/10-disable-stub.conf"
  mode = 420

  contents {
    source = "data:;base64,${base64encode(<<-EOT
      [Resolve]
      DNSStubListener=no
    EOT
    )}"
  }
}

systemd-resolved continues to handle resolution for the machine itself via /run/systemd/resolve/resolv.conf; it simply stops listening on port 53.

Next comes the configuration of Blocky and its systemd service. The files/blocky.yml file is injected using the templatefile() function, then mounted into the container within blocky.service:

data "ignition_file" "blocky_config" {
  path = "/etc/blocky/config.yml"
  mode = 420

  contents {
    source = "data:;base64,${base64encode(templatefile("${path.module}/files/blocky.yml", {
      dns_port  = local.blocky_dns_port
      http_port = local.blocky_http_port
    }))}"
  }
}

data "ignition_systemd_unit" "blocky" {
  name    = "blocky.service"
  enabled = true
  content = <<-EOT
    [Unit]
    Description=Blocky DNS
    After=docker.service network-online.target systemd-resolved.service
    [...]
    [Service]
    Restart=always
    RestartSec=5
    ExecStartPre=-/usr/bin/docker rm -f blocky
    ExecStartPre=/usr/bin/docker pull ${local.blocky_image}
    ExecStart=/usr/bin/docker run --name blocky --rm \
      -p ${local.blocky_dns_port}:${local.blocky_dns_port}/udp \
      -p ${local.blocky_dns_port}:${local.blocky_dns_port}/tcp \
      -p ${local.blocky_http_port}:${local.blocky_http_port}/tcp \
      -v /etc/blocky/config.yml:/app/config.yml:ro \
      ${local.blocky_image}
    [...]
  EOT
}

Before putting it all together, one last file is added: /etc/flatcar/update.conf. In the first part, the concept was explained: Flatcar automatically downloads its updates in the background and switches between partitions on reboot.

To prevent the DNS server from restarting in the middle of the day as soon as an update is ready, locksmithd is configured to allow an automatic restart only on Sunday mornings at 04:00, within a one-hour window:

data "ignition_file" "update_conf" {
  path      = "/etc/flatcar/update.conf"
  mode      = 420
  overwrite = true

  contents {
    source = "data:;base64,${base64encode(<<-EOT
      GROUP=${var.flatcar_channel}
      REBOOT_STRATEGY=reboot
      LOCKSMITHD_REBOOT_WINDOW_START=Sun 04:00
      LOCKSMITHD_REBOOT_WINDOW_LENGTH=1h
    EOT
    )}"
  }
}

Note the presence of overwrite = true on this file. This is because /etc/flatcar/update.conf already exists in the Flatcar base image. Without this attribute, Ignition would refuse to overwrite the existing file.

All that remains is to group all these building blocks together in the ignition_config data source:

data "ignition_config" "blocky" {
  users       = [data.ignition_user.core.rendered]
  directories = [data.ignition_directory.blocky.rendered]
  files       = [
    data.ignition_file.hostname.rendered,
    data.ignition_file.network.rendered,
    [...]
  ]
  links   = [data.ignition_link.timezone.rendered]
  systemd = [data.ignition_systemd_unit.blocky.rendered]
}

The data.ignition_config.blocky.rendered attribute now contains the complete JSON document, exactly like the one obtained using the butane binary in part one. You can also inspect an extract of the output using the tofu console command if required:

{
  "ignition": { "version": "3.5.0" },
  "passwd": {
    "users": [{ "name": "core", "sshAuthorizedKeys": ["ssh-ed25519 AAAA..."] }]
  },
  "storage": {
    "files": [
      {
        "contents": { "source": "data:;base64,YmxvY2t5Cg==" },
        "mode": 420,
        "path": "/etc/hostname"
      }
      [...]
    ]
  },
  "systemd": { "units": [...] }
}

As mentioned in the first blog post, this JSON will be evaluated atomically within the initramfs: if even a single directive is invalid, the machine will refuse to boot rather than end up in a partially configured state.

Getting ready for deployment
#

Everything is ready to create the virtual machine in main.tf. The first step is to inject the generated Ignition JSON as a snippet onto the Proxmox node, then associate it with the virtual machine.

Please note, however, that there are a few points to bear in mind regarding the code:

  • The first concerns boot-up: the Flatcar image requires UEFI. You must therefore switch the machine’s BIOS to ovmf with an EFI disk to avoid errors during boot;

  • The second relates to the system disk: it is imported directly from the image downloaded in the previous step. It is given a size larger than that of the image, and on first boot, Flatcar automatically extends its root partition to utilise the available space;

  • The final point is the bridge between Proxmox and Ignition. Proxmox attaches a small cloud-init drive to the virtual machine, and Flatcar’s Proxmox image can read the user-data to pass it on to Ignition. It is essential to establish this link via the initialization.user_data_file_id field within this resource.

But first, the configuration and secrets must be set up:

cp terraform.tfvars.example terraform.tfvars

export TF_VAR_proxmox_api_token="tofu@pve!provider=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export TF_VAR_encryption_passphrase="a-long-random-passphrase"

tofu init && tofu validate
tofu plan

Here is an overview of the main lines returned by tofu plan:

OpenTofu will perform the following actions:

  # proxmox_download_file.flatcar will be created
  + resource "proxmox_download_file" "flatcar" {
      + content_type = "import"
      + file_name    = "flatcar_production_proxmoxve_image-stable.qcow2"
      + url          = "https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_proxmoxve_image.img"
      [...]
    }

  # proxmox_virtual_environment_file.ignition will be created
  + resource "proxmox_virtual_environment_file" "ignition" {
      + content_type = "snippets"
      + source_raw {
          + file_name = "blocky-ignition.json"
          [...]
        }
    }

  # proxmox_virtual_environment_vm.blocky will be created
  + resource "proxmox_virtual_environment_vm" "blocky" {
      [...]
      + name    = "blocky"
      + vm_id   = 1000
      [...]
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Although rather lengthy, the plan is straightforward: these three resources are ready to be deployed.

$ tofu apply
[...]
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

blocky_api_url = "http://192.168.1.100:4000"
ssh_command = "ssh [email protected]"
vm_ip_address = "192.168.1.100"

In just a few minutes (especially the first time, when downloading the image), the virtual machine is created, started up and configured.

A few steps to check everything!
#

Once the instance has been deployed, you can check that everything is configured as intended! The best way to do this is to connect via SSH using ssh [email protected]:

$ cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID="flatcar"
ID_LIKE="coreos"
VERSION="4757.2.2"
[...]
$ sudo touch /usr/test
touch: cannot touch '/usr/test': Read-only file system

As you can see, it’s impossible to change anything in /usr, this is definitely the right OS. :-)

You can check the status of the update_engine service and the Blocky container using the relevant commands:

$ update_engine_client -status
CURRENT_OP=UPDATE_STATUS_IDLE
[...]

$ systemctl is-active blocky.service
active

$ docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
NAMES    IMAGE                           STATUS
blocky   ghcr.io/0xerr0r/blocky:v0.35.0  Up 2 minutes (healthy)

Outside the virtual machine, you can also test the DNS resolution:

$ dig +short @192.168.1.100 blog.filador.ch
188.114.96.12
[...]

$ dig +short @192.168.1.100 pagead2.googlesyndication.com
0.0.0.0

My blog resolves correctly, while the advertising domain returns 0.0.0.0 thanks to StevenBlack’s list. The DNS server is fully operational!

The cattle model in practice: modifying and recreating
#

What happens now if you want to change Blocky’s configuration, for example to add a new block list in files/blocky.yml?

You must always bear in mind Ignition’s golden rule: it runs only once, when the virtual machine is first started. If you simply edit the snippet file on Proxmox VE without recreating the virtual machine, absolutely nothing will happen, even if you restart the VM.

This is precisely the purpose of the lifecycle block placed at the end of the proxmox_virtual_environment_vm resource in main.tf:

  lifecycle {
    replace_triggered_by = [
      proxmox_virtual_environment_file.ignition,
    ]

    ignore_changes = [
      disk[0].import_from,
    ]
  }

Thanks to the replace_triggered_by directive, as soon as the content of the Ignition JSON changes (a modification to a list in files/blocky.yml, a change to the version of the Blocky image, or the addition of a systemd unit or an SSH key), the proxmox_virtual_environment_file.ignition resource is replaced, triggering the complete replacement of the virtual machine automatically during the tofu plan:

$ tofu plan
[...]
  # proxmox_virtual_environment_file.ignition must be replaced
-/+ resource "proxmox_virtual_environment_file" "ignition" {
      ~ id = "local:snippets/blocky-ignition.json" -> (known after apply)
      [...]
    }

  # proxmox_virtual_environment_vm.blocky will be replaced due to changes in replace_triggered_by
-/+ resource "proxmox_virtual_environment_vm" "blocky" {
      ~ id   = "1000" -> (known after apply)
        name = "blocky"
      [...]
    }

Plan: 2 to add, 0 to change, 2 to destroy.

This is a practical example of the cattle model: you never log in to the machine to edit /etc/blocky/config.yml manually and restart the service. You modify the single source of truth in Git, and OpenTofu destroys the old machine and replaces it within seconds with a brand-new, clean instance that is guaranteed to have no configuration drift.

Closing remarks
#

With just a few HCL files, you now have a Flatcar Container Linux virtual machine that is fully reproducible on Proxmox, deploying a functional Blocky DNS service without any manual intervention, all accompanied by automatic updates with a maintenance window.

You will find exactly the same philosophy and the same declarative discipline as those implemented with Talos Linux and Cluster API on Kubernetes clusters, but applied here to the infrastructure services that support them.

To take things further, I’d encourage you to look into extensions such as keepalived.sysext for a multi-machine cluster with a VIP set up to ensure high availability, or to use Flatcar as a replacement for Talos Linux, the choice is yours!

Flatcar Container Linux - This article is part of a series.
Part 1: Another step towards immutability with Flatcar Container Linux
Part 2: This Article

Related