The intersection of immutable infrastructure and lightweight orchestration represents a significant shift in how edge computing and bare metal deployments are managed. Flatcar Container Linux, an open-source project based on the CoreOS philosophy, provides a minimal, read-only root filesystem designed specifically to run containerized workloads. When paired with K3s, a stripped-down Kubernetes distribution optimized for edge environments, the result is a highly resilient, low-overhead cluster capable of operating in resource-constrained scenarios. Automating this synergy requires a deep understanding of the Ignition provisioning system and the specific constraints of the Flatcar environment, particularly concerning how the operating system handles write operations and how the K3s installation script interacts with the underlying filesystem.
Architectural Synergy of Flatcar and K3s
The combination of Flatcar Container Linux and K3s is engineered to eliminate the bloat associated with traditional Linux distributions and standard Kubernetes installations. K3s achieves its lightweight footprint by removing unnecessary cloud providers and replacing the etcd database with SQLite for single-node setups, though it supports highly available (HA) configurations for production environments.
Flatcar complements this by providing a container-optimized OS that prioritizes stability and security. Because Flatcar uses a read-only root filesystem, it prevents accidental configuration drift and ensures that the system remains in a known state. This architectural choice has direct implications for how K3s is installed; for example, standard installation paths that attempt to write to /usr/local/bin will fail on Flatcar, necessitating the use of specific directories like /opt/bin for binary placement.
Provisioning with Ignition and PXE Boot
Flatcar utilizes Ignition, a provisioning utility that allows for the configuration of the system during the first boot process. Unlike traditional configuration management tools that run after the OS is fully loaded, Ignition operates during the boot sequence to set up users, network settings, and initial system services.
The provisioning process typically involves two distinct Ignition files. The first file is utilized during the PXE (Preboot Execution Environment) boot process. This initial file is responsible for the primary installation of the OS onto the target disk.
A critical component of this initial stage is the installer.service. This systemd unit ensures that the Flatcar installation occurs correctly on the remote bare metal server.
```
[Unit]
Requires=network-online.target
After=network-online.target
[Service]
Type=forking
TimeoutStartSec=600
RemainAfterExit=yes
ExecStart=/usr/bin/sh -c "flatcar-install -d /dev/sda -i /opt/ignition.json && udevadm settle && systemctl reboot"
[Install]
WantedBy=multi-user.target
```
In this configuration, the flatcar-install command targets the /dev/sda drive and pulls the second Ignition file from a remote URL, such as http://10.0.0.5:8080/node-1-ignite-boot.ign. Once the installation is complete and the system reboots, the second Ignition file is used to configure the operational state of the node.
Automating K3s Server Installation
The automation of K3s on Flatcar is achieved by integrating the K3s installation script into the Ignition-defined systemd services. This allows the cluster to bootstrap itself without manual intervention following the OS installation.
To implement this, the K3s installation script must be delivered to the node. This is handled via the storage.files section of the Ignition configuration, which fetches the script from the official source.
yaml
storage:
files:
- path: /opt/k3s-install.sh
filesystem: root
mode: 777
contents:
remote:
url: https://get.k3s.io
Once the script is present on the disk, a dedicated systemd unit, k3s-install.service, is created to execute the installation. This service is designed to be idempotent, meaning it will not attempt to reinstall K3s if the binary already exists.
```
[Unit]
Description=Run K3s script
Wants = network-online.target
After = network.target network-online.target
ConditionPathExists=/opt/k3s-install.sh
ConditionPathExists=!/opt/bin/k3s
[Service]
Type=forking
TimeoutStartSec=180
RemainAfterExit=yes
KillMode=process
Environment="K3STOKEN=secretedgeclustertoken"
Environment="INSTALLK3S_EXEC=--cluster-init"
ExecStart=/usr/bin/sh -c "/opt/k3s-install.sh"
[Install]
WantedBy=multi-user.target
```
The use of ConditionPathExists=!/opt/bin/k3s is critical. It ensures that the installation script only runs if the K3s binary is not found in /opt/bin/k3s, preventing redundant installation cycles during subsequent boots.
Scaling the Cluster to Multiple Nodes
Expanding the cluster involves generating specific Ignition files for each subsequent node. While the base process remains identical to the first node, the configuration for the second and third nodes must be modified to join the existing cluster rather than initializing a new one.
For the primary server, the INSTALL_K3S_EXEC variable is set to --cluster-init. For subsequent nodes, this variable is adjusted to point to the existing server's API endpoint.
The configuration for the second node involves these environment variables:
K3S_TOKEN=secret_edgecluster_tokenINSTALL_K3S_EXEC='--server' 'https://10.0.0.70:6443'
This allows the second node to authenticate using the shared secret token and connect to the primary server at the specified IP address. The process is then repeated for the third node, ensuring the hostname is unique while maintaining the same cluster join parameters.
To finalize this deployment, the user must manage the PXE boot environment. This requires placing the Ignition files in /var/lib/tftp/ignition and mapping the corresponding PXE boot files in /var/lib/tftp/pxelinux.cfg to the IP addresses of the target nodes.
Technical Challenges and File System Constraints
Deploying K3s on Flatcar is not without its challenges, primarily due to the immutable nature of the operating system. A known issue occurs when the K3s installation script attempts to write to directories that are read-only.
Specifically, attempts to touch files in /usr/local/bin, such as /usr/local/bin/k3s-ro-test, result in a Read-only file system error. This is a direct consequence of Flatcar's security architecture. K3s solves this by installing binaries into /opt/bin, which is a writable area.
Another significant issue involves SELinux policies. In certain versions of K3s, such as v1.34.7+k3s1, the installation process may fail if the k3s-selinux policy is not found. The installer may prompt for the installation of container-selinux or a specific RPM from the Rancher repository.
[ERROR] Failed to find the k3s-selinux policy, please install:
yum install -y container-selinux
yum install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/k3s-selinux-1.6-1.el8.noarch.rpm
For Flatcar users, this creates a conflict because Flatcar does not use yum. Users have noted that the installation may break after specific Pull Requests (e.g., PR #13712), making it necessary to use specific versions of the install.sh script to maintain compatibility with Flatcar's immutable environment.
Infrastructure as Code with Ansible and Kube-vip
For those seeking a more abstracted approach than raw Ignition files, Ansible provides a powerful alternative for building HA clusters on Flatcar. This approach often integrates kube-vip to provide a virtual IP for the control plane, ensuring that the cluster remains available even if one of the master nodes fails.
The Ansible-based deployment is typically designed to be unopinionated, installing only the bare minimum requirements for a functional HA cluster.
| Feature | Standard K3s | Ansible-Flatcar HA Project |
|---|---|---|
| CNI | Included | Disabled |
| Ingress Controller | Traefik Included | Disabled |
| Load Balancer | ServiceLB Included | Disabled |
| High Availability | etcd / SQLite | kube-vip |
| Target OS | General Linux | Flatcar / Fedora CoreOS |
| Architectures | x86_64, ARM64 | x86_64, ARM64 |
The Ansible deployment requires Ansible version 2.4.0 or higher. By disabling the default K3s components like Traefik and ServiceLB, administrators can implement their own specific networking and ingress solutions that better suit their edge requirements.
Configuration Transpilation and Execution
To move from a human-readable YAML configuration to the JSON format required by Ignition, a transpiler tool (ct) is used. This is a mandatory step in the Flatcar provisioning workflow.
The command to convert a node's boot configuration is as follows:
ct < node-1-ignite-boot.yaml > node-1-ignite-boot.ign
This process transforms the declarative YAML definition into the structured JSON that the Flatcar bootloader can interpret. Once this file is generated and hosted on the TFTP server, the node can proceed through the PXE boot process, install the OS, and trigger the k3s-install.service to bootstrap the cluster.
Summary of Installation Parameters
The K3s installation on Flatcar depends heavily on environment variables passed through systemd. These variables dictate how the node behaves within the cluster.
K3S_TOKEN: A secret string used to authenticate nodes joining the cluster.INSTALL_K3S_EXEC: Determines if the node starts as a cluster initializer (--cluster-init) or joins an existing server (--server <URL>).K3S_VERSION: Specifies the version of K3s to be installed (e.g.,v1.34.7+k3s1).
When running manual installation tests on Flatcar, it is common to see complex execution strings to disable default components:
INSTALL_K3S_EXEC='server --tls-san 192.168.92.31 --disable=traefik,servicelb,local-storage,metrics-server --flannel-backend=none --disable-network-policy --disable-cloud-controller' INSTALL_K3S_VERSION='v1.34.7+k3s1' ./install.sh
This level of granularity allows the administrator to strip K3s down to its absolute core, reducing memory consumption and CPU overhead on edge hardware.
Analysis of High Availability and Stability
The stability of a K3s cluster on Flatcar is predicated on the successful hand-off between the Ignition provisioning and the systemd service execution. The primary point of failure in this architecture is typically the read-only nature of the root filesystem. If the K3s installation script is updated in a way that assumes a writable /usr/local/bin, the automation will fail.
Furthermore, the reliance on ConditionPathExists ensures that the system does not enter a boot loop by attempting to reinstall K3s on every restart. This is a critical stability feature for edge deployments where network connectivity may be intermittent, and nodes may reboot frequently due to power instability.
The integration of kube-vip in more advanced Ansible-driven deployments addresses the single point of failure inherent in a single-master setup. By providing a floating virtual IP, kube-vip ensures that kubectl commands and API requests are routed to a healthy master node, thereby achieving true high availability.
In conclusion, the automation of K3s on Flatcar Container Linux represents a sophisticated approach to edge infrastructure. By leveraging Ignition for the initial OS boot and systemd for the subsequent K3s orchestration, administrators can deploy clusters that are immutable, lightweight, and highly resilient. The key to success lies in managing the filesystem constraints of Flatcar and ensuring that the K3s installation variables are correctly mapped to the desired cluster topology.