Orchestrating Lightweight Kubernetes with K3s and Podman

The convergence of edge computing and automotive technology has necessitated a shift toward highly optimized, resource-efficient container orchestration. At the center of this shift is K3s, an official Cloud Native Computing Foundation project that provides a certified Kubernetes distribution stripped of the unnecessary dependencies and components found in vanilla Kubernetes (K8s). This lightweight architecture is critical for edge workloads where CPU, RAM, and storage are at a premium. When K3s is paired with SUSE Linux Enterprise, SUSE BCI, and Rancher, it forms a comprehensive ecosystem capable of powering software-defined vehicles. However, the introduction of an orchestration layer like K3s inevitably adds complexity to the software stack. For specific scenarios where a full Kubernetes distribution is overkill, Podman emerges as a powerful alternative or companion. Podman, which integrates seamlessly with systemd (the service manager used by SUSE), allows for the creation of containers, pods, and volumes based on Kubernetes YAML, albeit with certain limitations. Together, K3s and Podman create a versatile deployment strategy for in-vehicle and edge environments, balancing the power of a full orchestrator with the agility of a daemonless container engine.

The Architectural Synergy of K3s and Podman

K3s is designed specifically to reduce the overhead associated with standard Kubernetes. By removing unnecessary legacy providers and integrating critical components, it allows organizations to deploy Kubernetes-compatible environments in constrained spaces. While K3s handles the orchestration—managing the lifecycle, scaling, and networking of containers—Podman operates as the engine that manages the containers themselves.

Podman differs from Docker primarily in its daemonless architecture, which enhances security and reduces the attack surface of the host system. For k3d, a tool used to run K3s in Docker containers, Podman's Docker API compatibility layer is the bridge that allows k3d to function. k3d leverages the Docker API to provision and manage K3s nodes; since Podman v4 and higher provide an API compatible with Docker, k3d can target Podman as its backend. This creates a flexible stack where K3s provides the Kubernetes API and orchestration, while Podman manages the underlying container runtime.

Implementing k3d with Podman

Integrating k3d with Podman requires specific configurations to ensure the communication channels between the k3d CLI and the Podman socket are correctly established. Because k3d expects a Docker-like environment, the system must be told where to find the Podman socket.

Rootful Podman Configuration

In a rootful configuration, Podman runs with administrative privileges, mirroring the traditional Docker experience. The first requirement is ensuring the Podman system socket is active and available for k3d to connect to.

To enable the socket, the following command is used:

sudo systemctl enable --now podman.socket

Alternatively, if a systemd socket is not preferred, the socket daemon can be started manually:

sudo podman system service --time=0 &

A critical step for seamless integration is the handling of the service timeout. To prevent the Podman service from timing out during intensive K3s operations, a custom configuration file must be created in the containers configuration directory:

mkdir -p /etc/containers/containers.conf.d
echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf

To ensure k3d recognizes Podman as the primary engine without needing constant environment variable exports, a symbolic link can be created to map the Podman socket to the location where Docker is traditionally expected:

sudo ln -s /run/podman/podman.sock /var/run/docker.sock

Once the symbolic link is established or if the podman-docker package is installed, the cluster can be created simply via:

sudo k3d cluster create

If the user prefers not to use symbolic links, the DOCKER_HOST and DOCKER_SOCK environment variables must be explicitly defined:

export DOCKER_HOST=unix:///run/podman/podman.sock
export DOCKER_SOCK=/run/podman/podman.sock
sudo --preserve-env=DOCKER_HOST --preserve-env=DOCKER_SOCK k3d cluster create

Rootless Podman Configuration

Rootless Podman is highly preferred for security-sensitive environments, such as automotive systems, as it allows containers to run without root privileges. This minimizes the potential impact of a container breakout.

The first step is enabling the user-level Podman socket:

systemctl --user enable --now podman.socket

Or starting the service manually:

podman system service --time=0 &

For k3d to interact with the rootless socket, the XDG_RUNTIME_DIR must be correctly identified and exported to the environment:

XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
k3d cluster create

Cgroup v2 and Resource Delegation

Running Kubernetes (and by extension K3s) in a rootless environment introduces challenges regarding resource control. By default, non-root users are only granted delegation for the memory and pids controllers. However, K3s requires control over CPU, CPUSET, and I/O to function properly.

Verifying Cgroup Version

Before applying delegation settings, it is mandatory to verify that the system is running cgroup v2. This can be checked by looking for the presence of the cgroup.controllers file in the filesystem:

Check for: /sys/fs/cgroup/cgroup.controllers

If this file exists, the system is using cgroup v2. If it is absent, the system is using v1, and the following delegation steps will not apply as written.

Enabling Controller Delegation

To allow a non-root user to manage the necessary controllers, a systemd override must be created for the user service. This ensures that every time the user session starts, the correct controllers are delegated.

The following sequence of commands creates the necessary configuration:

mkdir -p /etc/systemd/system/[email protected]
cat > /etc/systemd/system/[email protected]/delegate.conf <<EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
systemctl daemon-reload

This configuration allows the user-session manager to delegate the specified controllers to the user, enabling the Kubelet and other K3s components to manage resource limits and quotas effectively within the rootless namespace.

Deployment on macOS

Deploying k3d with Podman on macOS requires an intermediate virtual machine, as Podman must run within a Linux kernel. The process involves initializing a Podman machine and configuring SSH access so k3d can communicate with the remote socket.

Machine Initialization and Connection

First, the Podman machine must be initialized and started:

podman machine init
podman machine start

Once the machine is running, the connection details must be retrieved to identify the correct URI and identity file:

podman system connection ls

This command will output a table including the name (e.g., podman-machine-default), the URI (e.g., ssh://core@localhost:53685/run/user/501/podman/podman.sock), and the Identity file path.

SSH Configuration

To allow k3d to use the Podman machine via SSH, the OpenSSH config file must be updated. This removes the need to manually pass identity files to every command.

vim ~/.ssh/config

Add the following block:

Host localhost
IdentityFile /Users/myusername/.ssh/podman-machine-default

Rootless vs Rootful macOS Execution

For rootless execution on macOS, the environment variables must point to the user-level socket:

export DOCKER_HOST=ssh://core@localhost:53685
export DOCKER_SOCKET=/run/user/501/podman/podman.sock
k3d cluster create --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@server:*'

The addition of the KubeletInUserNamespace feature gate is essential for K3s to operate correctly within the restricted user namespace provided by the Podman VM.

For rootful execution on macOS, the connection is mapped to the root user:

export DOCKER_HOST=ssh://root@localhost:53685
export DOCKER_SOCK=/run/podman/podman.sock
k3d cluster create

Podman Networking and Local Registries

Podman's networking model differs significantly from Docker's, particularly regarding the default "bridge" network. This creates specific hurdles when deploying k3d clusters and local registries.

DNS and Cluster Communication

The default Podman network has DNS disabled. Because K3s nodes rely on DNS for service discovery and inter-node communication, the default network is insufficient. A dedicated network with DNS enabled must be created:

podman network create k3d

To verify that DNS is indeed enabled on the new network:

podman network inspect k3d -f '{{ .DNSEnabled }}'

The output should return true. When creating the k3d cluster, it will now utilize this network to ensure stable internal communication.

Local Registry Management

When creating a local registry for k3d images, the --registry-create flag should be avoided. This is because --registry-create assumes the existence of a default "bridge" network, which Podman does not utilize in the same way as Docker.

Instead, the --default-network flag must be used to explicitly assign the registry to the Podman network:

k3d registry create --default-network podman mycluster-registry

To link this registry to a specific cluster during creation, use the --registry-use flag:

k3d cluster create --registry-use mycluster-registry mycluster

Advanced Rootless K3s Configuration

Running K3s in a fully rootless mode—outside of k3d—requires a different set of tools and configurations. Rootless K3s utilizes rootlesskit and slirp4netns to facilitate communication between the host network and the user network namespaces.

Systemd Integration for Rootless K3s

To run K3s as a rootless service, the following systemd commands are used:

systemctl --user daemon-reload
systemctl --user enable --now k3s-rootless

To verify the deployment and check the status of the pods:

KUBECONFIG=~/.kube/k3s.yaml kubectl get pods -A

Terminal Session Constraints

It is critical to note that running k3s server --rootless directly in a standard terminal session will fail. This is because terminal sessions do not allow cgroup v2 delegation by default. If a manual terminal execution is required, it must be wrapped in a systemd scope to grant the necessary permissions:

systemd-run --user -p Delegate=yes --tty k3s server --rootless

Rootless Network Tuning

The behavior of rootlesskit and slirp4netns can be tuned using environment variables defined in the Environment field of the k3s-rootless systemd unit.

The Role of CRI and Backend Runtimes

Understanding the relationship between K3s and Podman requires a deep look at the Container Runtime Interface (CRI). K3s and vanilla Kubernetes require a CRI-compatible backend to manage container lifecycles.

Historically, the Docker daemon provided two different endpoint interface sets from the same socket: one for the CLI client and one for the CRI. While Podman and CRI-O share the same image cache storage technology, Podman itself is not a CRI implementation.

Technically, it is possible to use CRI-O as the K3s backend. This involves setting up CRI-O independently on the host and then configuring the K3s server to point toward the CRI-O socket. While this is not directly supported as a "plug-and-play" feature, it allows for an architecture where K3s handles the orchestration and CRI-O handles the execution, all while maintaining compatibility with Podman's image management tools.

Technical Specification and Compatibility Summary

The following table summarizes the compatibility and requirements for integrating K3s, k3d, and Podman across different operating environments.

Conclusion: Analyzing the Edge Orchestration Landscape

The integration of K3s and Podman represents a strategic evolution in how containerized workloads are deployed in resource-constrained environments. By leveraging K3s, organizations can maintain the standard Kubernetes API—ensuring that the broader ecosystem of Helm charts, operators, and manifests remain usable—while stripping away the bloat of a traditional K8s distribution. The addition of Podman introduces a critical layer of security and flexibility. The daemonless nature of Podman, combined with the ability to run in rootless mode, directly addresses the security concerns inherent in automotive and industrial IoT deployments, where a compromised container must not be allowed to compromise the host OS.

The technical challenges associated with this stack—specifically regarding cgroup v2 delegation and Podman's networking model—highlight the complexity of moving away from the monolithic Docker daemon. The requirement for manual controller delegation (cpu, cpuset, io) underscores the strictness of the Linux kernel's security model in rootless environments. Furthermore, the necessity of creating custom Podman networks with DNS enabled for k3d demonstrates that while Podman is "Docker compatible," it is not a "Docker clone," and requires a more nuanced understanding of networking primitives.

Ultimately, the K3s-Podman combination is an ideal fit for "software-defined vehicles." In such a scenario, the vehicle can run a mix of critical and non-critical services. High-priority orchestration tasks can be handled by K3s, while lightweight, isolated utility containers can be managed directly by Podman via systemd. This hybrid approach optimizes for both reliability and security, providing a scalable foundation for the future of edge computing.

Sources

1. SUSE - Running Containerized Workloads in Vehicles
2. k3d - Podman Usage Documentation
3. k3s-io - GitHub Discussions
4. k3s - Advanced Documentation