K3S Infrastructure Orchestration on Proxmox Virtualization

The deployment of Kubernetes in a home laboratory environment requires a strategic balance between resource overhead and functional capability. Proxmox Virtual Environment provides a robust foundation for this through its support for both full Virtual Machines (VMs) and Linux Containers (LXC). K3S, a lightweight distribution of Kubernetes, is specifically engineered to reduce the footprint of the orchestration layer, making it the ideal candidate for Proxmox-based deployments. By integrating K3S within Proxmox, users can simulate production-grade environments—complete with multi-node clusters, load balancers, and service meshes—without requiring an enterprise-level hardware budget. This architecture allows for the experimentation of advanced networking concepts such as MetalLB for load balancing, NGINX for ingress traffic management, and Istio for a comprehensive service mesh.

Architectural Foundation and Resource Allocation

The success of a K3S deployment on Proxmox depends heavily on the initial resource allocation and the choice of virtualization technology. Depending on the goal, a user may opt for LXC containers for maximum efficiency or full VMs for maximum isolation.

LXC Container Configuration

Using LXC containers allows for a significantly lower overhead compared to full virtualization, as containers share the host's kernel. When deploying K3S on LXC, specific resource thresholds are recommended to ensure stability.

CPU Allocation: 2 CPU cores are typically assigned to each node. This ensures that the Kubernetes control plane and the kubelet have sufficient processing power to handle pod orchestration and health checks.
Memory Management: 4GB of RAM is the standard baseline. This prevents the node from entering an Out-of-Memory (OOM) state when running the K3S binary alongside essential system pods.
Disk Space: 32GB of disk space is allocated for the container image. This provides ample room for the OS, the K3S binary, and the container images pulled from registries.
Swap Space: No Swap is configured. Kubernetes generally performs poorly or fails to initialize properly when swap is enabled, as the kubelet requires precise memory accounting.
Container Type: Privileged containers are mandatory. K3S requires access to the host's kernel features and device nodes that are restricted in unprivileged containers.

Virtual Machine (VM) Specifications

For those requiring stronger isolation or specific kernel modifications, full VMs are the preferred route. This approach mimics a physical server environment more closely.

Disk Configuration: 100GB is recommended for VM disks, with write-back caching enabled to improve I/O performance.
CPU and Memory: At least 2 CPU cores are recommended for all nodes. For memory, a tiered approach is used: the Control Plane node receives 4GB of RAM, while worker nodes can be scaled down to 2GB if hardware resources are limited.
Guest Agent: The installation of the qemu guest agent is highly recommended. This allows Proxmox to communicate more effectively with the guest OS for tasks like graceful shutdowns and IP reporting.

Network Infrastructure and DNS Management

Kubernetes relies heavily on stable networking and DNS resolution for service discovery and ingress routing. In a Proxmox environment, this is handled through a combination of static IP assignments and local DNS zones.

IP Address Management

Consistency in network addressing is critical for the K3S cluster to maintain communication between the master and worker nodes.

Static IP Assignment: Every node in the cluster must be assigned a static IP address. This prevents the cluster from breaking if a DHCP server re-assigns addresses after a reboot.
DHCP Reservations: For users preferring centralized management, DHCP reservations (static leases) are configured in tools like Pi-hole to ensure the same IP is always delivered to the same MAC address.
Networking Defaults: Standard network settings are typically left as default within Proxmox, provided the static IP is correctly applied at the OS or DHCP level.

DNS Strategy and Service Discovery

To expose applications to the rest of the local network via an Ingress controller, a robust DNS setup is required.

Local DNS Zones: A primary zone, such as lab.local, is created on a DNS server (e.g., a Synology NAS).
A Record Mapping: A records are created for every K3S VM or LXC container. This allows the user to refer to nodes by hostname rather than IP address.
Hosts File Integration: For rapid testing or in environments without a centralized DNS server, mappings are added directly to the /etc/hosts file in dnsmasq. This ensures that application hostnames are correctly resolved to the internal cluster IP.

LXC Container Hardening and K3S Compatibility

Running K3S in an LXC container requires modifications to the container's configuration on the Proxmox host. Without these changes, K3S will fail to initialize due to missing permissions and restricted access to kernel subsystems.

Host-Level Configuration

The container must be stopped before the configuration file located at /etc/pve/lxc/<container_id>.conf can be edited. The following parameters must be added to the config file:

lxc.apparmor.profile: unconfined: This disables the AppArmor profile for the container, preventing it from blocking the system calls required by the K3S binary.
lxc.cgroup.devices.allow: a: This allows the container to access the necessary device nodes.
lxc.cap.drop:: This ensures that no capabilities are dropped, giving K3S the full range of permissions it needs to manage the node.
lxc.mount.auto: "proc:rw sys:rw": This mounts the proc and sys filesystems as read-write, which is essential for the K3S agent to manage kernel parameters.

Internal Container Tweaks

K3S expects the existence of /dev/kmsg for logging purposes, but LXC does not provide this by default. To resolve this, a script is added to /etc/rc.local within the container.

Implementation: The following command creates the script:
echo '#!/bin/sh -e ln -s /dev/console /dev/kmsg mount - make-rshared /' > /etc/rc.local
Execution: The script is made executable using chmod +x /etc/rc.local followed by a system reboot.
Persistence: This ensures that every time the container boots, the symlink to the console is created and the root filesystem is made shared, satisfying K3S requirements.

Operating System Provisioning and User Management

The choice of operating system provides the foundation for the K3S installation. Ubuntu is the primary recommendation across various implementation paths.

OS Installation Paths

LXC Templates: Ubuntu 20.04.1 LTS templates are commonly used for container-based setups due to their stability and compatibility.
VM Installations: Ubuntu 22.04 Server is recommended for VM-based setups. This involves a standard installation process including language selection, keyboard configuration, and the installation of the base server package.
Cloud-Init Integration: To streamline the creation of multiple nodes, Cloud-Init is utilized. This allows for the pre-configuration of usernames, passwords, and the injection of SSH public keys (e.g., ~/.ssh/id_ed25519.pub) before the VM is converted into a template.

User and Permission Setup

To avoid running the K3S process as the root user, a dedicated service user is created.

User Creation: A user named kubernetes is added to the system using the adduser kubernetes command.
Sudo Configuration: To allow this user to execute administrative tasks without constant password prompts, the /etc/sudoers file is modified via visudo to include:
kubernetes ALL=(ALL) NOPASSWD:ALL
Access: This setup allows the administrator to securely access the node via SSH using ssh kubernetes@kube-master.

K3S Installation and Cluster Initialization

The installation of K3S is designed to be a streamlined process. Once the OS is prepared and the environment is configured, the K3S binary is deployed.

Installation Process

The installation involves running the K3S installation script, which performs the following actions:

Binary Acquisition: The system downloads the latest stable release (e.g., v1.28.5+k3s1) from the official GitHub repository.
Verification: The downloaded binary is verified using a SHA256 checksum to ensure integrity.
Installation: The binary is installed to /usr/local/bin/k3s.
Symlink Creation: The installer creates symlinks for essential tools:
- kubectl: The primary Kubernetes CLI tool.
- crictl: The Container Runtime Interface CLI.
- ctr: The containerd CLI.
Helper Scripts: Utility scripts such as /usr/local/bin/k3s-killall.sh and /usr/local/bin/k3s-uninstall.sh are generated to facilitate cluster maintenance.

Service Integration

K3S integrates with the system's init system to ensure the cluster starts automatically upon boot.

Systemd Integration: The installer creates a systemd service file at /etc/systemd/system/k3s.service and an environment file at /etc/systemd/system/k3s.service.env.
Service Activation: The unit is enabled and started using systemd, which initiates the K3S process.
Kubeconfig Export: The cluster configuration is saved to a kubeconfig file. To interact with the cluster from a remote workstation, the following commands are used:
export KUBECONFIG=/Users/andrew/kubernetes/kubeconfig
kubectl config use-context default
kubectl get node -o

Advanced Cluster Expansion and Services

Once the base K3S installation is complete, the cluster can be expanded and enhanced with production-like services.

Multi-Node Scaling

Scaling the cluster involves adding worker nodes to the initial master node.

VM Cloning: Using Proxmox's cloning feature, a template (created from the first VM) is used to create multiple clones (e.g., k3s-00 through k3s-04). Full clones are preferred over linked clones for better performance and independence.
Node Join: Worker nodes join the cluster using the token provided by the master node, expanding the total compute capacity available for pods.

Networking and Mesh Implementation

To move beyond a basic installation, several networking components are integrated:

MetalLB: This provides a network load balancer for the cluster, allowing services to be exposed via a single virtual IP.
NGINX Ingress Controller: This manages external access to services within the cluster, routing traffic based on hostnames and paths.
Istio Service Mesh: For complex microservices architectures, Istio is installed to provide advanced traffic management, security, and observability.
Kubernetes Dashboard: A web-based UI is deployed and exposed via an Ingress to provide a visual overview of the cluster's health and resource usage.

Cluster Maintenance and Reliability

Maintaining a K3S cluster on Proxmox requires a proactive approach to backups and state management.

Baseline Backups

A critical step in the deployment process is taking a backup of the container or VM immediately after the OS and basic permissions are configured but before K3S is installed.

Baseline Utility: These backups serve as a "known good" state. If the K3S installation fails or the configuration becomes corrupted, the user can revert to the baseline in seconds.
Template Generation: Backups of the master node can be used to rapidly deploy identical worker nodes, eliminating the need to repeat manual configuration steps for every new node.

Storage Integration

For persistent data, K3S can be integrated with external storage solutions.

Synology Integration: By using the open-source Synology CSI plugin for Kubernetes, the cluster can leverage a Synology NAS (e.g., DS 723+) for persistent volume claims.
CSI Plugin: The Container Storage Interface (CSI) allows Kubernetes to dynamically provision and manage storage on the Synology device, ensuring that data persists even if a pod is rescheduled to a different node.

Conclusion

The deployment of K3S on Proxmox represents a sophisticated convergence of virtualization and container orchestration. By carefully selecting between LXC and VM architectures, administrators can tune their environment for either maximum density or maximum isolation. The process requires meticulous attention to the underlying system permissions—particularly the Apparmor and cgroup settings in LXC—to ensure the K3S binary can operate without restriction.

The integration of a dedicated DNS strategy, utilizing either a Synology NAS or Pi-hole, transforms a collection of isolated nodes into a cohesive, reachable network. When combined with tools like MetalLB, NGINX, and Istio, the resulting infrastructure is not merely a home lab but a high-fidelity replica of a production Kubernetes environment. This setup empowers users to master the complexities of microservices, service meshes, and persistent storage in a safe, scalable, and highly flexible environment.