K3s on Fedora CoreOS

The convergence of Fedora CoreOS (FCOS) and K3s represents a paradigm shift in how container orchestration is deployed on bare metal and virtualized environments. Fedora CoreOS provides a bulletproof, minimal host system characterized by atomic updates, which ensures that the underlying operating system remains consistent and predictable. By integrating K3s—a lightweight, certified Kubernetes distribution—users can deploy applications using industry-standard tools such as kubectl and helm without the overhead associated with traditional Kubernetes installations. This combination is particularly potent for self-hosting services, such as RSS readers, or deploying edge computing solutions where resource efficiency and system stability are paramount. The architecture leverages the immutability of CoreOS to create a foundation that is resistant to configuration drift, while K3s provides the agility of a single-binary orchestration layer.

Architecture of K3s and Fedora CoreOS

K3s, developed by Rancher, is specifically engineered for edge and IoT devices, though its utility extends to home servers and production environments. The primary architectural distinction of K3s is its lightweight nature; it is packaged as a single binary that is less than 40MB in size.

One of the most significant design choices in K3s is the replacement of the heavyweight etcd store. In standard Kubernetes distributions, etcd is the primary data store, which can be resource-intensive. K3s utilizes an embedded sqlite database to store its data, which significantly reduces memory and CPU consumption. While sqlite is the default, other persistence backends can be configured if the specific use case requires them.

Fedora CoreOS complements this by providing an immutable root filesystem. The OS separates immutable system data from mutable application data. Specifically, all application-owned data, including configuration files and databases, is stored in /var. This separation is critical during system re-installs or updates, as it allows the system image to be replaced without losing the state of the applications running atop the cluster.

CoreOS Configuration and Ignition Logic

Configuring Fedora CoreOS requires a departure from traditional manual installation. Instead, it utilizes "Butane" and "Ignition" files. Ignition files are JSON documents that manifest the various configurations and tasks the system must perform during the initial boot-up.

The process of deploying K3s on FCOS involves a multi-stage approach due to the immutable nature of the OS. The general workflow follows these stages:

• Install the OS on the machine's hard disk, followed by a reboot.
• Install the required RPM package for K3s to be installable, followed by a reboot.
• Install the K3s binary and initialize the cluster.

This staged process is necessary because certain configurations require the system to be in a specific state before the next layer of software can be applied. To automate this, developers use nested Ignition files and systemd services, which are then baked into a custom CoreOS ISO image. This allows any machine booting from the ISO to automatically install both CoreOS and K3S without manual intervention.

Bare Metal Deployment and ISO Customization

For bare metal installations, the objective is to transform a physical computer into a fully functional, single-node Kubernetes cluster via a USB stick. This process involves the use of several specialized tools.

The required toolset for this workflow includes:

• coreos-installer: Used to install the OS onto the hardware.
• butane: Used to convert human-readable configuration into Ignition JSON.
• ignition-validate: Used to ensure the Ignition file is syntactically correct.
• rpm-ostree: Used for managing packages on the immutable host.

A critical warning for bare metal installation is that booting from the customized USB stick will wipe all existing data on the machine. This process occurs without a confirmation prompt, meaning the installation begins immediately upon boot.

To embed the configuration into the ISO image, the following podman command is utilized:

bash podman run --privileged --pull=always --rm -v .:/data -w /data quay.io/coreos/coreos-installer:release iso ignition embed -i coreos-autoinstall.ign ./fedora-coreos-32.20201004.3.0-live.x86_64.iso

In this command, the coreos-autoinstall.ign file is embedded into the ISO. Once the machine boots from this image, it automatically installs CoreOS and K3s. Users can then access the machine via SSH using the keys defined in the FCC file. The Kubernetes control plane configuration (kubeconfig) is located at /etc/rancher/k3s/.

Manual Installation and Virtualization

While automation via ISO is ideal for scaling, manual installation is often used in virtualization environments. This process requires specific steps to ensure the environment is prepared for the K3s binary.

The installation sequence is as follows:

First, enter a root shell:

bash sudo -i

Next, install the necessary SELinux package for K3s:

bash rpm-ostree install https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm

After the package installation, the system must be rebooted to apply the changes:

bash systemctl reboot

Once the system is back online, the K3s installation is triggered using the following commands:

bash sudo -i export K3S_KUBECONFIG_MODE="644" export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik" curl -sfL https://get.k3s.io | sh -

The use of INSTALL_K3S_EXEC allows the user to disable the default deployment of ServiceLB and Traefik, providing more control over the networking stack. After installation, the status of the service and the health of the nodes can be verified:

bash systemctl status k3s kubectl get nodes -o wide kubectl get pods -A -o wide

Expanding the Cluster: Joining Worker Nodes

K3s allows for the expansion of a single-node cluster into a multi-node environment by adding worker nodes. This requires the extraction of a join token from the master node.

To retrieve the join token, execute:

bash sudo cat /var/lib/rancher/k3s/server/node-token

An example of a join token would be: K1042e2f8e353b9409472c1e0cca8457abe184dc7be3f0805109e92c50c193ceb42::node:c83acbf89a7de7026d6f6928dc270028.

On the worker node, the following configuration is applied before running the installation script:

bash export K3S_KUBECONFIG_MODE="644" export K3S_URL="https://k3s-master:6443" export K3S_TOKEN="K1042e2f8e353b9409472c1e0cca8457abe184dc7be3f0805109e92c50c193ceb42::node:c83acbf89a7de7026d6f6928dc270028" curl -sfL https://get.k3s.io | sh -

Data Persistence and System Re-installs

One of the primary challenges in immutable infrastructure is preserving data during system updates or re-installs. Fedora CoreOS addresses this by mapping /usr/local to a subvolume in /var.

Because /var is designed to be mutable, it can be preserved during a re-install. This is configured in the Butane (server.bu) file. The following configuration specifies the disk partitions and ensures the filesystem is not wiped:

yaml storage: disks: - device: /dev/disk/by-id/coreos-boot-disk wipe_table: false partitions: - number: 4 label: root size_mib: 8192 resize: true - label: var size_mib: 0 filesystems: - path: /var device: /dev/disk/by-partlabel/var format: xfs wipe_filesystem: false with_mount_unit: true

In this configuration, wipe_filesystem: false is the key directive that prevents the loss of application data. Furthermore, because the K3s binary is installed in /usr/local/bin, it persists across re-installs. However, to ensure the binary is updated to the latest version during a re-install, the overwrite: true setting is used in the Ignition configuration. This ensures that the old binary is replaced by the new version while the /var data remains intact.

Networking and Security Considerations

Integrating advanced networking plugins like Cilium on Fedora CoreOS can introduce complexities, particularly regarding SELinux.

Technical reports indicate that Cilium may fail to operate on Fedora CoreOS 40 if SELinux is enabled. While Cilium documentation suggests that the plugin should work with SELinux provided that envoy.enabled=false is set during installation, users have reported continued failures.

A successful workaround identified in community discussions involves setting the security context to privileged mode:

bash --set securityContext.privileged=true

This adjustment allows the Cilium agent to function correctly on Fedora CoreOS 40, although the underlying reason for the SELinux conflict remains a subject of investigation. This highlights the necessity of carefully managing security contexts when deploying CNI (Container Network Interface) plugins on immutable hosts.

Technical Specifications Comparison

The following table summarizes the key technical attributes of the K3s on FCOS stack.

Advanced Post-Installation Possibilities

Once the base K3s cluster is operational on Fedora CoreOS, the system can be expanded with several high-level configurations to increase utility and resilience.

The following enhancements are possible:

• Use of netboot to provision machines instead of utilizing the coreos-installer ISO.
• Implementation of an automatic updates strategy to keep the host OS current.
• Configuration of an ad-blocking DNS server utilizing dnsmasq.
• Deployment of a VPN using wireguard for secure remote access to the cluster.
• Creation of a highly available (HA) Kubernetes control plane to eliminate single points of failure.
• Installation of Fedora CoreOS on Raspberry Pi hardware, although this requires additional documentation-specific steps.

Analysis of the FCOS-K3s Ecosystem

The synergy between Fedora CoreOS and K3s creates a robust environment that bridges the gap between the stability of an appliance and the flexibility of a cloud-native platform. The primary strength of this architecture lies in the "atomic" nature of the host. By treating the operating system as a versioned image rather than a collection of installed packages, the risk of "configuration drift"—where two nodes in a cluster slowly diverge in their settings—is virtually eliminated.

The use of SQLite in K3s is a critical optimization for this environment. In a bare-metal scenario, especially on low-power hardware, the memory overhead of etcd often becomes a bottleneck. By utilizing sqlite, K3s allows the host to allocate more resources to the actual application workloads rather than the orchestration overhead.

However, the transition to this model requires a steep learning curve. The shift from manual configuration to Ignition-based provisioning means that administrators must think in terms of "desired state" rather than "sequential steps." The nested Ignition approach described—where the system reboots multiple times to apply different layers of configuration—is a sophisticated way to handle the constraints of an immutable filesystem.

From a security perspective, the interaction between SELinux and CNI plugins like Cilium demonstrates that the "bulletproof" nature of FCOS can sometimes clash with the privileged requirements of low-level networking tools. The necessity of securityContext.privileged=true suggests that while the OS is secure, the abstraction layer of Kubernetes occasionally requires "breaking out" of those constraints to manage network traffic at the kernel level.

Ultimately, the FCOS-K3s stack is an ideal choice for users who require a "set it and forget it" infrastructure. The combination of atomic updates, /var persistence, and a lightweight orchestration binary results in a system that is rock solid for long-term home server use or professional edge deployments.

Sources

1. DevNonsense
2. DevOpsTales
3. Murillo Digital
4. Fedora Project Discussion