K3S Proxmox LXC Orchestration

The deployment of Kubernetes in a homelab or edge environment often presents a paradox: the desire for the orchestration power of a production-grade system versus the limited resource availability of consumer-grade hardware. Kubernetes, while revolutionary, is notorious for being resource-heavy, often requiring significant overhead for its control plane and node agents. This is where k3s, the lightweight Kubernetes distribution engineered by Rancher, transforms the landscape. By stripping out legacy APIs, bundling containerd, and shipping as a single binary under 100MB, k3s provides a streamlined path to container orchestration. When integrated with Proxmox Virtual Environment (PVE), specifically utilizing Linux Containers (LXC), the efficiency is amplified. LXC allows for the deployment of Kubernetes nodes that share the host kernel, eliminating the need for a full guest operating system for every node. This architectural choice reduces memory consumption and CPU overhead, allowing users to bootstrap multi-node clusters on hardware that would otherwise struggle to support traditional virtual machines.

The process of integrating k3s with Proxmox LXC is not without its challenges. While a standard Virtual Machine (VM) provides a complete hardware abstraction, LXC containers operate under stricter security constraints and shared kernel limitations. These restrictions often clash with the requirements of k3s, which expects certain kernel capabilities, filesystem permissions, and device access to manage pods and networking effectively. To achieve a stable deployment, one must move beyond basic installation and dive into the underlying Linux internals. The journey involves configuring AppArmor profiles, adjusting cgroup permissions, and ensuring the existence of specific kernel message devices. By navigating these technical hurdles, an operator gains a critical layer of operational intuition. Rather than relying on the abstractions provided by managed cloud providers like AWS, where platform teams handle the scheduling and networking, building a cluster on Proxmox allows a technician to break things, debug them, and fundamentally understand why failures occur.

Architectural Comparison: LXC versus Virtual Machines

When designing a Kubernetes cluster on Proxmox, the primary decision lies in the choice between Virtual Machines (VMs) and LXC containers. While both can host k3s, their resource footprints and performance characteristics differ wildly.

The impact of these differences is most evident in resource-constrained environments. Because an LXC container shares the host kernel, it avoids the overhead of running a complete OS kernel in memory. This efficiency is a game-changer for homelabs, as it allows for the deployment of more worker nodes on the same physical hardware. Furthermore, the fast startup time of LXC containers is critical for ephemeral worker nodes or scenarios involving autoscaling, where the time to bring a node online directly impacts the availability of the application. From a storage perspective, LXC utilizes the host filesystem directly via a root filesystem overlay, which bypasses the virtual disk emulation layer used by VMs, resulting in I/O performance that is closer to bare metal.

Node Provisioning and Base Configuration

The foundation of a k3s cluster on Proxmox begins with the creation of the container instances. Whether the goal is a single-node setup or a multi-node cluster involving a mix of VMs and LXC containers, the initial provisioning must be precise to avoid downstream failures.

The following specifications are recommended for K3S nodes to ensure sufficient overhead for both the system and the deployed workloads:

• CPU: 2 CPU cores
• Disk: 32GB disk space
• RAM: 4GB
• Swap: No Swap
• Container Type: Privileged container
• IP Assignment: Static IP addresses

Using a privileged container is a mandatory requirement because k3s needs to perform operations that unprivileged containers are restricted from doing. Static IP addresses are essential for cluster stability; assigning these IPs to a DNS server's static records or mapping them in a dnsmasq /etc/hosts file ensures that application hostnames and services exposed via an ingress remain reachable.

Once the containers are created using a template such as Ubuntu 20.04.1 LTS, the base operating system must be prepared. The initial steps involve updating the package repository and installing essential tools:

apt update && apt upgrade && apt install curl

To adhere to security best practices and avoid running the orchestration software as the root user, a dedicated user should be created:

adduser kubernetes

To enable this user to perform administrative tasks without the friction of constant password prompts, the sudoers file must be modified using visudo to include the following entry:

kubernetes ALL=(ALL) NOPASSWD:ALL

Overcoming LXC Constraints for k3s

The most complex aspect of running k3s in Proxmox LXC is the reconciliation of LXC's security model with k3s's operational requirements. By default, Proxmox drops several capabilities from LXC containers that are vital for k3s, specifically CAP_SYS_ADMIN, CAP_NET_ADMIN, and CAP_SYS_PTRACE.

To resolve these issues, the container must be stopped to allow modifications to the configuration file located at /etc/pve/lxc/<container_id>.conf on the Proxmox host. The following configuration lines must be appended to the file:

lxc.apparmor.profile: unconfined lxc.cgroup.devices.allow: a lxc.cap.drop: lxc.mount.auto: "proc:rw sys:rw"

The impact of these changes is profound. Setting the AppArmor profile to unconfined removes the security restrictions that would otherwise block k3s from managing the container environment. Allowing cgroup devices and dropping the capability restrictions ensures that the k3s agent can interact with the kernel as required. The lxc.mount.auto directive ensures that the /proc and /sys filesystems are mounted as read-write, which is necessary for the k3s agent to monitor and manage system resources.

In addition to these configuration changes, k3s requires the /dev/kmsg device to exist within the container. Since LXC does not provide this by default, a custom script must be implemented in /etc/rc.local within the container to create a symbolic link from the console to the kernel message device:

echo '#!/bin/sh -e ln -s /dev/console /dev/kmsg mount - make-rshared /' > /etc/rc.local chmod +x /etc/rc.local reboot

Furthermore, the Proxmox container settings must be adjusted to enable specific kernel features:

• keyctl=1: This enables the Linux kernel keyring. This is a critical requirement for containerd, which uses the keyring to securely store credentials and keys necessary for pulling images from registries.
• nesting=1: This enables nested containerization. Because k3s runs containerd inside the LXC container, and containerd subsequently runs pods (which are themselves containers), nesting must be enabled. Without this, Proxmox will block the creation of inner containers, causing the pods to fail.

After these modifications, the container must be fully restarted using the following commands on the Proxmox host:

pct stop <container_id> pct start <container_id>

K3s Installation and Cluster Bootstrapping

With the environment properly configured, the installation of k3s is a streamlined process. For a master node running in a VM or an LXC container, the installation is triggered by a single command that automates the setup of the systemd service, the installation of containerd, and the bootstrapping of the Kubernetes cluster.

curl -sfL https://get.k3s.io | sh -

Immediately following installation, a common point of failure occurs when attempting to interact with the cluster. Running the following command:

kubectl get nodes

Typically results in the error: The connection to the server localhost:8080 was refused. This happens because kubectl defaults to localhost:8080 when no configuration is provided. k3s, however, stores its kubeconfig file at /etc/rancher/k3s/k3s.yaml. The user must point kubectl to this file or export the KUBECONFIG environment variable to establish a successful connection to the API server.

To verify the status of the k3s agent and ensure that the server is active and the kubelet has started, the following command can be used to monitor the logs:

journalctl -u k3s-agent -f

Multi-Node Cluster Expansion

A robust deployment often involves a hybrid approach to nodes to balance reliability and efficiency. A sample architecture involves one control plane (master) node and two worker nodes:

• Master Node: A Proxmox VM running Ubuntu. This provides a stable, isolated environment for the control plane, ensuring that the cluster management is not affected by LXC-specific quirks.
• Worker 1: A standard Proxmox VM.
• Worker 2: A Proxmox LXC container.

The LXC worker node allows the cluster to scale without consuming excessive RAM. Once the master node is operational, worker nodes can be joined to the cluster using the server URL and the node token provided by the master.

To avoid repeating the complex LXC configuration steps for every new worker node, it is recommended to take a backup of the first successfully configured LXC container. This backup serves as a baseline, allowing the operator to clone new worker nodes quickly while retaining all the necessary AppArmor, cgroup, and /dev/kmsg settings.

Advanced Services and Ecosystem Integration

Once the vanilla multi-node cluster is running stably—with k3s managing scheduling, networking, and DNS via CoreDNS—the infrastructure can be extended with production-grade tools to handle traffic and service discovery.

The intended roadmap for a fully realized k3s environment includes the following components:

• MetalLB: This provides a network load balancer for Kubernetes, allowing services to be exposed via a stable IP address on the local network.
• Ingress Controllers: Either NGINX or the Gateway API controller can be implemented to manage external access to services and handle routing based on hostnames.
• Istio Service Mesh: For advanced traffic management, mutual TLS (mTLS), and observability, Istio can be installed to manage the communication between microservices.
• Kubernetes Dashboard: A web-based UI can be deployed to provide a visual representation of the cluster state, accessible via an Ingress.

These components transform a basic k3s install into a powerful platform for experimenting with microservices architecture. By deploying sample applications onto this infrastructure, users can test how scheduling and networking behave under the hood, filling the knowledge gap that exists when using abstracted cloud environments.

Conclusion: Analytical Synthesis of the LXC-K3S Synergy

The integration of k3s within Proxmox LXC containers represents a strategic optimization for technical enthusiasts and homelab operators. The primary value proposition is the drastic reduction in resource overhead. When compared to traditional VM deployments, the shift to LXC results in a memory reduction of approximately 50% per node (from 500MB+ down to 250-300MB) and a significant acceleration in deployment speed. This efficiency allows for the creation of denser clusters on modest hardware, enabling the testing of complex distributed systems that would otherwise be cost-prohibitive.

However, the technical cost of this efficiency is the loss of the "black box" simplicity provided by VM virtualization. The necessity of modifying AppArmor profiles, enabling kernel keyring access via keyctl, and managing nested containerization via nesting=1 forces the operator to interact directly with the Linux kernel's security and resource management layers. This process, while potentially frustrating, is precisely what builds operational intuition. It transforms the act of deploying a cluster from a simple "copy-paste" exercise into a deep-dive study of how Kubernetes interacts with the host operating system.

The stability of the resulting cluster—composed of a control plane VM and a mix of VM and LXC workers—demonstrates that the perceived incompatibility between LXC's restrictive environment and k3s's requirements is not an absolute barrier, but rather a configuration challenge. By applying the minimal necessary changes to unblock the k3s agent, the operator maintains a reasonable security posture while gaining the performance benefits of containerization. Ultimately, the k3s-on-Proxmox-LXC model serves as a powerful bridge between the ease of Docker Compose and the complexity of production Kubernetes, providing a scalable, efficient, and educationally rich environment for exploring emerging technologies.

Sources

1. Running k3s on Proxmox - A Multi-Node Cluster with a VM and LXC Worker
2. From Zero to k3s on Proxmox LXC Part 1