Synchronizing Podman and K3s for Edge and Automotive Software Architectures

The modernization of software distribution in constrained environments, particularly within the automotive sector and edge computing nodes, requires a departure from the heavyweight nature of traditional Kubernetes distributions. The intersection of Podman and K3s represents a strategic pivot toward lightweight, daemonless, and rootless container orchestration. While vanilla Kubernetes (K8s) provides a robust framework for data center operations, its dependencies and component overhead make it impractical for resource-constrained environments like in-vehicle compute platforms. K3s, an official Cloud Native Computing Foundation (CNCF) project, addresses this by stripping away unnecessary components, delivering a certified Kubernetes distribution that maintains the power of the K8s API while minimizing the footprint. When integrated with SUSE Linux Enterprise, SUSE BCI, and Rancher, K3s forms a comprehensive ecosystem capable of managing the complex lifecycles of software-defined vehicles.

However, the orchestration layer is only one half of the equation. The container runtime—the engine that actually executes the images—must be equally efficient. This is where Podman emerges as a critical contender. Unlike traditional Docker-based setups, Podman is daemonless and designed to integrate seamlessly with systemd, the standard service manager used by SUSE. This synergy allows for a more secure, granular approach to container management, particularly when rootless operation is required to minimize the attack surface of the host system. While K3s provides the orchestration (the "brain"), Podman provides the execution (the "muscle"), creating a combination that is uniquely suited for the stringent requirements of automotive and edge deployments.

The Architecture of K3s and the Role of Podman

K3s is engineered specifically for the edge. By reducing the number of dependencies compared to vanilla Kubernetes, K3s allows organizations to deploy a fully functional K8s environment on hardware that would otherwise be unable to support the overhead. This is vital for in-vehicle workloads where CPU and memory are earmarked for critical safety functions and infotainment systems rather than cluster management. Despite its lightness, K3s remains a certified Kubernetes distribution, ensuring that any workload designed for standard K8s can be ported to the vehicle with minimal friction.

Podman complements this architecture by offering a flexible alternative to the Docker daemon. In many scenarios, the added complexity of a full Kubernetes orchestration layer may be overkill. Podman allows operators to create containers, pods, and volumes using Kubernetes-compliant YAML files, albeit with some limitations compared to a full K3s deployment. This means a developer can use Podman for local development or simple deployments and then scale to K3s for complex, multi-node orchestration without changing the core definition of the application.

Implementing k3d with Podman Backend

The tool k3d serves as a wrapper to simplify the creation of K3s clusters. While k3d is fundamentally designed around the Docker API, it maintains a compatibility layer for Podman v4 and higher. It is important to note that Podman support within k3d is currently classified as experimental, meaning that while functional, it does not carry the same stability guarantees as the Docker backend.

Connection Configurations for Rootless and Rootful Modes

To utilize Podman as the backend for k3d, the environment must be told where to find the Podman socket, as k3d expects a Docker-compatible API.

For rootless mode, the following environment variables must be exported to point k3d toward the user-specific Podman socket:

DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock

Once these variables are set, the cluster can be initialized using:

k3d cluster create

In scenarios where rootful mode is required, the connection strings change to target the system-wide Podman socket:

export DOCKER_HOST=ssh://root@localhost:53685
export DOCKER_SOCK=/run/podman/podman.sock

Following the export of these variables, the cluster is created via:

k3d cluster create

macOS Integration and SSH Configuration

Running Podman on macOS requires a virtual machine, as Podman is native to Linux. The initialization process involves setting up a podman machine to act as the backend.

The setup sequence begins with:

podman machine init

Or, if a machine already exists:

podman machine start

To obtain the necessary connection details, the user must run:

podman system connection ls

This command reveals the URI and Identity file associated with the machine, such as ssh://core@localhost:53685/run/user/501/podman/podman.sock. To ensure k3d can connect to this remote socket, the OpenSSH config file must be updated. This is done by editing the file:

vim ~/.ssh/config

And adding the following entry to map the identity file:

Host localhost IdentityFile /Users/myusername/.ssh/podman-machine-default

Advanced Networking and Registry Management in Podman

One of the primary differences between Podman and Docker is the approach to networking. Podman does not utilize a default "bridge" network in the same way Docker does, which introduces specific challenges for k3d cluster communication and registry creation.

DNS and Network Creation

The default Podman network has DNS disabled, which prevents k3d cluster nodes from resolving addresses. To resolve this, a dedicated network with DNS enabled must be created manually.

The following steps ensure connectivity:

podman network create k3d

To verify that DNS is enabled on the newly created network, run:

podman network inspect k3d -f '{{ .DNSEnabled }}'

The output should return true.

Local Registry Implementation

When creating local registries with k3d while using Podman, the absence of a default bridge network means the user cannot rely on the --registry-create flag, as that flag assumes a bridge network exists. Instead, the --default-network flag must be used to explicitly specify the Podman network.

To create a registry, use:

k3d registry create --default-network podman mycluster-registry

To subsequently link this registry to a specific cluster during creation, use the --registry-use flag:

k3d cluster create --registry-use mycluster-registry mycluster

System-Level Configuration and Cgroup v2 Delegation

For Podman and K3s to operate correctly in a rootless environment, the underlying Linux kernel must properly delegate control groups (cgroups). Specifically, cgroup v2 is required to allow non-root users to manage resources like CPU and I/O.

Verifying Cgroup Version

The presence of the following file indicates that cgroup v2 is active:

/sys/fs/cgroup/cgroup.controllers

If this file is missing, the system is running cgroup v1, which will limit the capabilities of rootless container orchestration.

Enabling Resource Delegation

By default, non-root users are only granted delegation for memory and PIDs controllers. For a robust K3s or Podman environment, CPU, CPUSET, and I/O delegation must be enabled. This is achieved by creating a systemd configuration drop-in file.

First, create the directory:

mkdir -p /etc/systemd/system/[email protected]

Then, create the delegation configuration file:

bash cat > /etc/systemd/system/[email protected]/delegate.conf <<EOF [Service] Delegate=cpu cpuset io memory pids EOF

Finally, reload the systemd daemon to apply the changes:

systemctl daemon-reload

Rootless K3s Deployment and Management

Rootless K3s is a specialized configuration that allows the Kubernetes server to run without root privileges, significantly increasing the security posture of the host. This is achieved through the use of rootlesskit and slirp4netns, which handle the communication between the host network namespace and the user network namespace.

Execution and Initialization

Running K3s in rootless mode should not be attempted directly in a standard terminal session, as terminals do not support cgroup v2 delegation. Instead, the process should be managed via systemd.

To initialize a rootless K3s instance:

systemctl --user daemon-reload
systemctl --user enable --now k3s-rootless

To verify that the pods are operational, use the following command:

KUBECONFIG=~/.kube/k3s.yaml kubectl get pods -A

If a terminal execution is absolutely necessary for testing, the command must be wrapped in a systemd scope to ensure delegation:

systemd-run --user -p Delegate=yes --tty k3s server --rootless

Rootless Configuration Variables

The behavior of rootlesskit and slirp4netns can be tuned using environment variables. These should be added to the Environment field of the k3s-rootless systemd unit.

The Interface Layer: CRI and Runtime Compatibility

Understanding the relationship between K3s and Podman requires a deep dive into the Container Runtime Interface (CRI). The CRI is the standardized interface that K8s and K3s use to communicate with the underlying container runtime.

Historically, the Docker daemon provided two different endpoint interface sets from a single socket: the CRI for the orchestrator and the CLI for the user. Podman and CRI-O utilize the same image cache storage technology. While K3s is designed to work with specific runtimes, it is technically possible—though not directly supported—to use CRI-O as the K3s backend. This requires the user to manually set up CRI-O and then point the K3s server toward the CRI-O socket.

Podman Desktop and Developer Experience

Beyond the command line, Podman Desktop provides a graphical interface for managing containers and Podman machines. This is particularly useful for developers who are transitioning from Docker Desktop to an open-source, rootless alternative.

Podman Desktop offers deep integration with modern development tools. For instance, it integrates seamlessly with the Visual Studio Code Dev Containers extension, allowing developers to define their development environment in a container and launch it directly from the IDE. For those managing home labs or testing custom workloads, Podman Desktop serves as a centralized hub to manage containers without the overhead of a full Kubernetes cluster, while still maintaining the ability to deploy to a K3s environment once the application is ready for orchestration.

Strategic Analysis of the Podman-K3s Ecosystem

The integration of Podman and K3s is more than a technical convenience; it is a strategic architectural choice for the next generation of embedded and edge computing. The shift toward "Software-Defined Vehicles" (SDVs) demands a system that is modular, secure, and lightweight.

By utilizing K3s, automotive manufacturers can implement a standardized Kubernetes API, enabling the use of Helm charts and standard K8s manifests to manage vehicle services. This avoids vendor lock-in and allows for a vast ecosystem of cloud-native tools to be used on the edge. When paired with Podman, the security profile is heightened. The removal of a central, root-privileged daemon eliminates a primary attack vector. If a containerized application is compromised, the attacker is trapped within a rootless user namespace, preventing them from gaining root access to the underlying vehicle operating system.

Furthermore, the ability to use Podman for simple, single-container services and K3s for complex, inter-dependent microservices allows for a tiered deployment strategy. For example, a simple logging agent might run as a standalone Podman container managed by systemd, while the infotainment orchestration and OTA (Over-the-Air) update manager run as a K3s cluster. This hybrid approach optimizes resource utilization and reduces the complexity of the software stack.

The learning curve for this setup is primarily centered around the networking and cgroup configuration. The transition from the "everything is a bridge" mentality of Docker to the explicit network management of Podman requires a more disciplined approach to infrastructure as code. However, the payoff is a system that is significantly more resilient and aligned with the security requirements of critical infrastructure.

Sources

1. SUSE - Running Containerized Workloads in Vehicles
2. k3d - Advanced Podman Usage
3. K3s GitHub Discussions
4. Podman Desktop
5. K3s Advanced Documentation