The convergence of Proxmox Virtual Environment and K3S represents a paradigm shift for home lab enthusiasts and DevOps engineers seeking to mirror production-grade Kubernetes environments without the overhead of massive enterprise hardware. By leveraging Proxmox as the underlying virtualization layer, users can abstract physical hardware into flexible, scalable virtual machines that serve as the foundation for a K3S cluster. K3S, a lightweight Kubernetes distribution engineered by the Rancher team, strips away legacy cloud providers and unnecessary components, making it an ideal candidate for resource-constrained environments. The synergy between these two technologies allows for the creation of multi-node clusters that support advanced networking, persistent storage via external plugins, and rapid reproducibility. Whether the goal is to learn the intricacies of container orchestration, test microservices architecture, or build a resilient home server, the combination of Proxmox and K3S provides a scalable architecture that can grow from a single-node setup to a sprawling multi-node cluster across several physical hosts.
Proxmox Virtualization Foundation
The first critical step in establishing a Kubernetes environment is the installation of Proxmox VE on a physical machine. For those operating on a budget, using refurbished hardware, such as Lenovo Thinkcentre micro desktops, provides a cost-effective way to build a robust homelab.
The installation process begins with downloading the Proxmox-VE ISO and creating a bootable USB drive. During the boot process, the guided wizard handles the installation, with the most critical configurations being the selection of a hostname and the assignment of a static IP address. Once the installation is complete, Proxmox provides a URL to a web management console, allowing the administrator to manage the entire host through a browser.
The use of Proxmox provides several operational advantages:
- Resource Abstraction: Physical hardware is partitioned into Virtual Machines (VMs), allowing for the precise allocation of CPU, RAM, and storage.
- Snapshotting and Templates: Proxmox allows for the creation of VM templates, which significantly reduces the time required to deploy new nodes.
- Hardware Flexibility: The system can run on various hardware configurations, from high-end workstations to micro-desktops.
Virtual Machine Provisioning and Cloud-Init
To avoid the tedious process of installing an operating system manually for every Kubernetes node, Proxmox offers integration with Cloud-Init. This allows for the automated configuration of VM settings, including users, passwords, and SSH keys.
The provisioning workflow involves adding an Ubuntu Server image via the local storage interface. Under the ISO Images section, the image can be downloaded directly from a URL. Once the image is present, the Cloud-Init interface is used to define the identity of the VM. This includes the addition of a username, a password, and the pasting of an SSH public key, such as the contents of ~/.ssh/id_ed25519.pub.
To ensure scalability, the initial VM is converted into a template. From this template, multiple VMs can be created using the Full Clone mode. For instance, a five-node cluster can be deployed by cloning the template and naming the VMs sequentially, such as k3s-00 through k3s-04.
Network Configuration and DNS Strategy
Kubernetes requires stable networking to maintain communication between the control plane and the worker nodes. There are two primary strategies for managing IP addresses in a Proxmox K3S environment: static IP assignment and DHCP reservations.
Static IP addresses are often assigned during the Ubuntu Server installation process. However, an alternative approach is using DHCP leases via a server like Pi-hole to ensure each VM consistently receives the same IP address.
For DNS management, integrating a Synology DS 723+ or similar NAS can boost productivity. By installing the DNS server from the Synology Package Center, a primary zone can be created, such as lab.local. Within this zone, A records are added for each K3S VM, ensuring that nodes can resolve each other by name rather than relying solely on IP addresses. This is essential for maintaining a manageable cluster as it scales.
Ubuntu Server Configuration Specifications
The choice of operating system is pivotal for cluster stability. Ubuntu 22.04 Server is frequently utilized as the base OS for K3S VMs due to its reliability and wide community support.
When configuring the VMs in Proxmox, specific resource allocations are recommended to ensure optimal performance:
- Disk Space: 100GB is recommended, with the write-back feature enabled for improved I/O performance.
- CPU Cores: A minimum of 2 cores per VM is recommended to handle the container orchestration overhead.
- Memory (RAM): The Control Plane node requires at least 4GB of RAM to manage the cluster state, while worker nodes can operate with 2GB if system memory is limited.
- Guest Agent: Installing the qemu guest agent is highly recommended after the OS is running to improve communication between the Proxmox host and the VM.
The installation process for Ubuntu Server involves selecting the preferred language and keyboard layout, opting for the base installation, and configuring the network settings to the assigned static IP address.
K3S Cluster Deployment
K3S is a minimalistic Kubernetes distribution that simplifies the installation process. The deployment varies depending on whether the node is the initial control plane or a subsequent worker node.
For the first node (the control plane), the installation is straightforward. Running the installation command handles the setup and ensures that the K3S service starts automatically upon rebooting the VM.
For subsequent worker nodes, the process requires connecting the new VM to the existing cluster. This is achieved by setting two specific environment variables:
K3S_URL: The URL of the control plane node.K3S_TOKEN: The security token used to authenticate the worker node.
Once these variables are set, the installation command is run, and the worker node joins the cluster. Success is confirmed once the join process is completed, and the cluster can thereafter be managed via the web UI or the command line.
Infrastructure as Code (IaC) and Automation
For users prioritizing reproducibility, manual installation is replaced by automation tools. This allows for the destruction and rebuilding of the cluster to facilitate experimentation.
A professional automation stack for Proxmox K3S includes:
- Terraform: Used for the initial provisioning of the virtual machines. Terraform handles the infrastructure layer, although it is noted that it does not track configuration changes as effectively as configuration management tools.
- Ansible: Used for the software configuration and K3S deployment. Ansible playbooks, such as
setup-k3s.yml, automate the installation across all nodes.
The reproducible execution flow for this stack is as follows:
- Run
terraform initto initialize the provider. - Run
terraform applyto provision the VMs. - Navigate to the ansible directory:
cd ansible. - Execute the playbook:
ansible-playbook -i hosts setup-k3s.yml. - Set the Kubeconfig:
export KUBECONFIG=./k3s.yaml. - Verify nodes:
kubectl get nodes.
Advanced Orchestration and Management
Once the K3S cluster is operational, further tools can be integrated to enhance management and functionality.
Helm is typically installed to manage Kubernetes applications through charts. To provide a graphical user interface for the cluster, Rancher can be deployed. While some prefer the CLI for its efficiency, Rancher provides a comprehensive visual representation of the cluster's health and resources.
For storage, the Synology CSI (Container Storage Interface) plugin can be leveraged, allowing the Kubernetes cluster to use the Synology NAS for persistent volumes. This prevents data loss when pods are rescheduled across different nodes.
Hardware and Resource Specifications
The following table summarizes the recommended resource allocations for a Proxmox-based K3S cluster.
| Component | CPU Cores | RAM | Disk Space | OS |
|---|---|---|---|---|
| Control Plane | 2+ | 4GB | 100GB | Ubuntu 22.04 Server |
| Worker Node | 2 | 2GB | 100GB | Ubuntu 22.04 Server |
Troubleshooting and System Integrity
Maintaining the health of a K3S cluster requires attention to both the orchestration layer and the underlying services.
One critical area is the database configuration. If an external database is used for the cluster state, it must be properly configured and accessible from the control plane nodes. Failure to ensure connectivity results in errors within the systemctl status k3s output, specifically complaining that the connection to the MySQL box was closed or rejected.
From a security perspective, particularly in production environments, the following practices are essential:
- Secret Management: Never store passwords in clear text; use a vault for production secrets.
- Privilege Escalation: Remove sudo privileges from automation users (e.g.,
ansiblebot) once the provisioning process is complete. - Network Security: Ensure that internal cluster communication is isolated from public networks.
Analysis of Deployment Methodologies
The deployment of K3S on Proxmox reveals a clear trade-off between manual control and automated scalability.
The manual method, involving Proxmox web UI and individual VM configuration, is ideal for beginners. It allows users to understand each layer of the stack, from the hypervisor to the Kubernetes API. However, this method is prone to human error and is difficult to replicate.
The automated approach, leveraging Terraform and Ansible, transforms the infrastructure into code. This methodology allows for "disposable infrastructure," where a cluster can be torn down and restarted daily. This is particularly valuable in lab environments where experimentation often leads to configuration drift or system instability.
The integration of Cloud-Init further accelerates this process by removing the need for manual OS installation. By creating a gold image and converting it to a template, the time to deploy a new node is reduced from hours to seconds.
Finally, the use of an external DNS server and DHCP reservations solves the common "node-not-found" errors associated with dynamic IP addressing. While some argue that static IPs are for the lazy and not suitable for cloud deployments, in a local Proxmox environment, they provide the stability necessary for a reliable Kubernetes control plane.