K3S and Fedora CoreOS Integration

The marriage of K3S and Fedora CoreOS (FCOS) represents a convergence of two philosophy-driven technologies: the lightweight, certified Kubernetes distribution and the immutable, atomic host operating system. K3S, engineered by Rancher, is designed specifically for edge computing and Internet of Things (IoT) environments where resource constraints are a primary concern. It achieves this by packaging the entire distribution into a single binary of less than 40MB, stripping away the heavyweight etcd store typically found in production-grade Kubernetes and replacing it with an embedded SQLite database. This architectural shift significantly reduces memory overhead and simplifies the deployment process. Fedora CoreOS complements this by providing a bulletproof, minimal host system that utilizes atomic updates. This ensures that the underlying infrastructure remains consistent and predictable, while K3S provides the orchestration layer necessary to deploy applications using standard tools such as kubectl and helm.

The synergy between these two technologies allows for the creation of a robust, self-hosting environment capable of running everything from simple RSS readers to complex microservices architectures. Because FCOS is immutable, it requires a specific approach to installation and configuration, moving away from traditional imperative commands toward a declarative model. The integration utilizes Ignition files—JSON documents that define the system's desired state—and Butane, a tool used to convert human-readable YAML configurations into those JSON Ignition files. By baking these configurations into a custom ISO image, administrators can achieve a "zero-touch" deployment where the OS and the Kubernetes cluster are provisioned automatically upon booting.

Architectural Foundation of K3S

K3S is a certified Kubernetes distribution that prioritizes efficiency without sacrificing the API compatibility of standard Kubernetes. The primary objective of K3S is to provide a production-ready environment that can run on low-power hardware.

• Single Binary Distribution: K3S is packaged as a single binary, which simplifies the installation process and reduces the attack surface of the host.
• SQLite Persistence: In a departure from standard Kubernetes, K3S uses an embedded SQLite database for data storage instead of etcd. This reduces the resource requirements for the control plane, although other persistence backends can be configured if specific project requirements demand them.
• Resource Optimization: The distribution is designed to be less than 40MB, making it ideal for edge devices where storage and RAM are limited.

The impact of this architecture is that it enables the deployment of production-grade orchestration on hardware that would traditionally be unable to support the overhead of a full Kubernetes installation. By eliminating the complexity of etcd for single-node or small-cluster deployments, K3S lowers the barrier to entry for home servers and IoT gateways.

Fedora CoreOS Immutable Infrastructure

Fedora CoreOS is designed for the container-first era, providing a minimal operating system that is updated atomically. This means the OS is not modified in place; instead, it is updated by replacing the entire system image.

• Atomic Updates: FCOS uses a system of atomic updates to ensure that the host remains in a known good state. If an update fails, the system can roll back to the previous version.
• Immutability: The core system files are read-only, which prevents configuration drift and increases the security posture of the machine.
• Ignition Configuration: FCOS relies on Ignition files to handle the initial configuration of the system during the first boot. This includes setting up users, disks, and network configurations.

The consequence of this immutability is that traditional installation methods—such as manually running a series of bash scripts—are discouraged. Instead, the system uses a declarative approach. This ensures that every node in a cluster is identical, which is critical for maintaining stability in a Kubernetes environment.

Deployment Strategies for Bare Metal and Virtualization

Depending on the environment, the installation of K3S on Fedora CoreOS varies. Whether deploying on bare metal via a USB stick or within a virtualization environment, the goal is to reach a functional state where kubectl can be used to manage the cluster.

Virtualization Environment Installation

In virtualized environments, the installation follows a more direct path, often involving the manual installation of required dependencies.

1. Enter a root shell using sudo -i.
2. Install the required SELinux package for K3S via rpm-ostree. The specific package used is https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm.
3. Perform a system reboot to apply the changes.
4. Execute the K3S installation script. This involves setting specific environment variables to configure the installation.
   • export K3S_KUBECONFIG_MODE="644" is used to ensure the kubeconfig file has appropriate permissions.
   • export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik" is used to disable the default ServiceLB and Traefik ingress if they are not required.
5. Run the installation command: curl -sfL https://get.k3s.io | sh -.
6. Verify the installation using systemctl status k3s, kubectl get nodes -o wide, and kubectl get pods -A -o wide.

Bare Metal Automated Installation

For bare metal deployments, a more sophisticated approach is used to avoid manual intervention. This involves creating a custom ISO image that automates the entire process.

• Nested Ignition Files: Due to the immutable nature of CoreOS, the installation happens in stages. This requires nested Ignition files to allow the machine to reboot and apply configurations incrementally.
• Custom ISO Production: The process involves creating a custom ISO image using podman and the coreos-installer image.
• Automation Workflow:
  1. Install the OS on the hard disk and reboot.
  2. Install the RPM package required for K3S and reboot.
  3. Install the K3S binary.

To implement this, a run-coreos-installer systemd service is used. This service runs the CoreOS installer and passes the K3S Ignition file as an argument. A critical detail in this process is the ExecStartPost action; in some configurations, this performs a poweroff rather than a reboot to ensure the ISO image is unmounted, preventing the installer from looping.

The final step to bake this into an ISO involves the following command:
podman run --privileged --pull=always --rm -v .:/data -w /data quay.io/coreos/coreos-installer:release iso ignition embed -i coreos-autoinstall.ign ./fedora-coreos-32.20201004.3.0-live.x86_64.iso

Data Persistence and System Reinstalls

One of the primary challenges of using an immutable OS is preserving application data during system reinstalls or updates. FCOS solves this by separating immutable system data from mutable application data.

• The /var Directory: All application-owned data, including databases and configurations, is stored in /var. This directory is mutable and can be preserved across reinstalls.
• Disk Configuration: To ensure /var is preserved, the Ignition configuration must explicitly define the storage layout.

The following table describes the storage configuration required to preserve data:

Furthermore, Fedora CoreOS maps /usr/local to a subvolume in /var. Since the K3S binary is installed in /usr/local/bin, it persists across reinstalls. To ensure the binary is updated to the latest version during a reinstall, the overwrite: true flag is set in the server.bu configuration.

Cluster Expansion and Node Management

Once the initial K3S server is operational on FCOS, the cluster can be expanded by adding worker nodes. This transforms a single-node instance into a distributed Kubernetes cluster.

Joining Worker Nodes

To join a new node to the existing cluster, a join token from the server node is required.

1. Retrieve the join token from the master node:
   sudo cat /var/lib/rancher/k3s/server/node-token
2. On the worker node, set the following environment variables:
   • export K3S_KUBECONFIG_MODE="644"
   • export K3S_URL="https://k3s-master:6443"
   • export K3S_TOKEN="[TOKEN_FROM_MASTER]"
3. Execute the installation script:
   curl -sfL https://get.k3s.io | sh -

Advanced Configuration Options

Beyond the basic installation, several advanced configurations can be implemented to increase the utility and resilience of the cluster.

• High Availability (HA): The control plane can be configured for high availability to prevent a single point of failure.
• Networking: Users can implement various CNI (Container Network Interface) plugins. Options include Calico and Cilium, which can be installed using k3sup.
• Security: K3S can be secured using gVisor to provide stronger isolation for containers.
• Network Services: The host can be configured with dnsmasq for ad-blocking DNS or wireguard for secure VPN access.
• Provisioning: Instead of using ISO images, netboot can be used to provision machines.

Technical Specifications Summary

The following table summarizes the core components and requirements for a K3S on FCOS deployment.

Detailed Analysis of the Integration

The integration of K3S and Fedora CoreOS provides a blueprint for modern edge infrastructure. The most significant achievement of this pairing is the elimination of the "configuration drift" problem. In traditional Kubernetes deployments on mutable operating systems, each node can slowly deviate from the original configuration as manual changes are made over time. By using FCOS, the host is treated as an appliance. If a node becomes unstable, it is not repaired; it is simply reprovisioned using the same Ignition files, ensuring that the state is restored to exactly what was defined in the code.

The use of SQLite instead of etcd is not merely a space-saving measure; it is a strategic choice for the edge. etcd is notoriously sensitive to disk latency and network instability, which are common in IoT environments. SQLite provides a more resilient, albeit less scalable, alternative for small-scale clusters. This makes the K3S on FCOS stack particularly effective for "homelab" enthusiasts and industrial edge deployments where the cluster size remains small but the need for reliability is high.

Furthermore, the separation of /var from the root filesystem allows for an elegant upgrade path. When the K3S binary needs to be updated, the administrator can trigger a system reinstall. Because the application data in /var is preserved and the binary in /usr/local/bin is overwritten, the upgrade process is atomic. This removes the risk of partial updates or corrupted configurations that often plague imperative update processes.

In conclusion, the combination of K3S and Fedora CoreOS transforms a standard piece of hardware into a production-grade orchestration platform. The reliance on declarative configuration through Butane and Ignition, coupled with the resource efficiency of K3S, creates a system that is both easy to deploy and simple to maintain. While the initial setup of nested Ignition files and custom ISOs involves a steeper learning curve than a simple script, the resulting infrastructure is significantly more robust and scalable.

Sources

1. Murillo Digital
2. DevNonsense
3. DevOpsTales