k3d and the Architecture of K3s within Docker Containers

The intersection of container orchestration and container runtime environments has reached a point of high efficiency with the introduction of k3d. At its core, k3d is a specialized program engineered to run a K3s cluster within Docker. K3s itself is a lightweight, CNCF-certified Kubernetes distribution and a Sandbox project. Because K3s is designed specifically for low-resource environments, it is distributed as a single binary that operates using under 512MB of RAM. The primary function of k3d is to leverage Docker images built from the K3s repository to deploy multiple K3s nodes as Docker containers on any host machine where Docker is already installed.

The architectural significance of this approach is that it decouples the Kubernetes control plane and worker nodes from the underlying physical hardware or virtual machine. This allows a single physical or virtual machine, referred to as the Docker Host, to simultaneously support multiple K3s clusters. Each of these clusters can maintain its own set of server and agent nodes, providing an isolated environment for various projects, tests, or development stages. This eliminates the need for multiple virtual machines to simulate a multi-node cluster, significantly reducing the overhead on the host system.

The Mechanics of K3s and Docker Integration

The integration of K3s into Docker is achieved through the use of specialized images provided by Rancher. These images allow the K3s server and agent to run as standard Docker containers. For those who wish to bypass the k3d wrapper and use the docker run command directly, the rancher/k3s image is the primary vehicle.

To launch a K3s server via Docker, a specific command structure is required to ensure the container has the necessary permissions and network visibility. The following command illustrates the deployment:

sudo docker run --privileged --name k3s-server-1 --hostname k3s-server-1 -p 6443:6443 -d rancher/k3s:v1.24.10-k3s1 server

The use of the --privileged flag is critical here, as it grants the container the necessary permissions to manage the container runtime and network interfaces required by Kubernetes. The -p 6443:6443 mapping ensures that the Kubernetes API is accessible from the host machine.

A critical technical requirement when using the rancher/k3s image is the specification of a valid K3s version as the tag. The latest tag is not maintained, meaning users must explicitly state the version they intend to deploy to ensure stability and compatibility. Furthermore, Docker image tagging conventions prohibit the use of the plus sign (+) in tags. Consequently, a hyphen (-) must be used in the tag instead to comply with Docker Hub requirements.

Once the K3s server is active and running within the Docker container, the administrator must obtain the admin kubeconfig to interact with the cluster. This is achieved by copying the configuration file from the container to the host system using the following command:

sudo docker cp k3s-server-1:/etc/rancher/k3s/k3s.yaml ~/.kube/config

This action ensures that kubectl on the host machine can communicate with the K3s API server running inside the container.

SELinux Support and System Security

When deploying K3s on systems where Security-Enhanced Linux (SELinux) is enabled by default, such as CentOS, specific policy configurations are required to prevent the security layer from blocking K3s operations. Proper SELinux policies must be installed to ensure that the K3s binary and its associated containers can execute the required system calls.

There are two primary methods for handling SELinux policies:

• Automatic Installation: The install script provided by Rancher is designed to automatically install the SELinux RPM from the Rancher RPM repository. This occurs automatically as long as the system is compatible and the installation is not being performed in an air-gapped environment.
• Manual Installation: In cases where automatic installation is not possible, such as air-gapped systems, users must manually install the required SELinux policies to ensure cluster stability.

k3d Functional Capabilities

k3d provides a sophisticated management layer over the raw Docker containers, allowing for the lifecycle management of K3s clusters without requiring manual docker run commands for every node. As of version v4.0.0, released in January 2021, k3d offers a comprehensive suite of capabilities.

The primary capabilities of k3d include:

• Cluster Lifecycle Management: Users can create, stop, start, and delete entire K3s clusters. Additionally, k3d allows for the growing or shrinking of clusters by adding or removing individual nodes.
• Configuration Flexibility: These operations can be performed via command line flags for quick tasks or via a configuration file for reproducible environment definitions.
• Registry Management: k3d can manage and interact with container registries, which are essential for pulling the images that will run within the K3s cluster.
• Kubeconfig Management: The tool automates the management of Kubeconfigs, ensuring that the host's kubectl is always pointed at the correct cluster.
• Image Importation: k3d enables the import of images from the local Docker daemon directly into the container runtime that is running inside the cluster, bypassing the need to push images to a remote registry during development.

Use Cases for k3s in Docker

The dominant use case for k3d is local development on Kubernetes. The primary objective is to provide developers with a tool that allows them to run a lightweight Kubernetes cluster on their local machine with minimal hassle and low resource consumption.

By providing a production-like environment locally, developers can achieve fast iteration times. This is a significant improvement over the traditional approach of running docker-compose locally and then deploying to a Kubernetes cluster in a separate environment. By using k3d, the development environment mimics the production architecture (K3s/Kubernetes), reducing the "it works on my machine" discrepancy.

Deployment Methodologies in k3d

k3d allows for different levels of complexity depending on the needs of the user, ranging from a "simple" one-command setup to a "sophisticated" multi-node configuration.

The Simple Deployment

For users who need a basic environment quickly, k3d provides a streamlined creation process. The following command is used:

k3d cluster create

When this command is executed, k3d spawns a K3s cluster consisting of two containers: a Kubernetes control-plane node (server) and a load balancer (serverlb) positioned in front of it. These containers are placed within a dedicated Docker network. The Kubernetes API is then exposed on a randomly chosen free port on the Docker host. Additionally, a named Docker volume is created in the background, which serves as the preparation area for future image imports.

If no name is provided during the creation process, k3d assigns the default name k3s-default. The resulting containers follow a specific naming convention:

• The load balancer is named k3d-k3s-default-serverlb.
• The first server node is named k3d-k3s-default-server-0.

Following the creation, k3d automatically pulls the Kubeconfig from the cluster and merges it with the user's default Kubeconfig, typically located at $HOME/.kube/config or the path specified by the KUBECONFIG environment variable.

The Sophisticated Deployment

For developers who require a high-availability (HA) setup or a specific number of worker nodes, k3d supports advanced configuration via command line flags. An example of such a command is:

k3d cluster create mycluster --api-port 127.0.0.1:6445 --servers 3 --agents 2 --volume '/home/me/mycode:/code@agent[*]' --port '8080:80@loadbalancer'

This command results in the creation of six containers:

• 1 load balancer: Handles incoming traffic and distributes it to the servers.
• 3 servers: These act as the control-plane nodes, providing redundancy for the cluster management.
• 2 agents: These are the worker nodes (formerly referred to as worker nodes) that run the actual application workloads.

The --api-port 127.0.0.1:6445 flag tells k3d to map the internal Kubernetes API port (6443) to the host's port 6445. This allows the user to define exactly how they connect to the cluster API. The --volume flag allows for the mounting of local host directories into the agent nodes, while the --port flag maps host ports to the load balancer.

Operational Management and Monitoring

Once a cluster is deployed, k3d provides several tools for interaction and monitoring. To verify the status of the nodes in the cluster, the standard kubectl command is used:

kubectl get nodes

Beyond kubectl, k3d provides its own set of management commands to list the various components of the environment:

• k3d cluster list: Lists all currently created clusters.
• k3d node list: Lists the nodes within the clusters.
• k3d registry list: Lists the registries managed by k3d.

For users seeking help with the command line interface, k3d includes a built-in help system accessible via:

k3d [command] --help

Technical Specifications Comparison

The following table summarizes the key components and specifications of the K3s in Docker ecosystem.

Detailed Analysis of k3d Infrastructure

The implementation of k3d represents a shift in how local Kubernetes development is handled. By utilizing Docker containers to wrap K3s, k3d effectively creates a "cluster-in-a-box" experience. This architecture allows for the simulation of complex multi-node topologies on a single machine, which was previously only possible through the use of resource-heavy virtual machines.

The "Deep Drilling" of the k3d architecture reveals that the tool is not merely a script to run Docker containers, but a management layer that handles the intricacies of Kubeconfig merging, network orchestration, and volume management. The ability to import images from the local Docker daemon is particularly significant; it eliminates the latency and bandwidth costs associated with pushing images to a registry during the development cycle.

From a DevOps perspective, k3d provides an ideal environment for testing CI/CD pipelines. Since k3d clusters are "throwaway," they can be spun up in a GitHub Action or GitLab CI runner, tested, and then destroyed in seconds. This aligns with the modern Infrastructure as Code (IaC) philosophy, where environments are ephemeral and reproducible.

Furthermore, the support for multiple servers (control-plane nodes) via k3d allows developers to test High Availability configurations locally. Testing how an application behaves when a control-plane node fails is critical for production stability, and k3d makes this scenario easy to replicate by simply increasing the --servers count.

Conclusion

The deployment of K3s within Docker, facilitated by k3d, provides a powerful solution for local Kubernetes development and testing. By reducing the RAM requirement to under 512MB and leveraging the efficiency of Docker containers, k3d removes the traditional barriers to entry for Kubernetes. The ability to manage the entire lifecycle of a cluster—from creation and scaling to deletion—through a single CLI tool ensures that developers can focus on their applications rather than the underlying infrastructure.

The technical requirements, such as the use of specific version tags for rancher/k3s images and the avoidance of the plus sign in Docker tags, highlight the necessity of adhering to Docker's strict tagging conventions. Similarly, the integration of SELinux policies ensures that the system remains secure without compromising the functionality of the cluster.

Ultimately, the transition from using docker-compose to k3d allows for a more accurate representation of production environments. The capacity to map API ports, mount local volumes, and manage load balancers ensures that the local development experience is as close to the final deployment as possible. As the industry continues to move toward lightweight, edge-compatible Kubernetes distributions, the synergy between K3s and Docker via k3d will remain a critical asset for developers and DevOps engineers alike.

Sources

1. Introduction to k3d: Run K3s in Docker
2. K3d (K3s in Docker)
3. K3s Advanced Documentation
4. Docker Hub: rancher/k3s