Orchestrating Secure Containerized Pipelines via Podman Integration within GitLab CI

The architectural paradigm of Continuous Integration and Continuous Deployment (CI/CD) is undergoing a fundamental shift away from monolithic, daemon-dependent container engines toward decentralized, daemonless, and rootless containerization. For years, the industry standard for executing containerized workloads within CI/CD pipelines has relied heavily on Docker-in-Docker (DinD). While functional, DinD introduces significant security vulnerabilities and operational complexities, primarily because it necessitates running containers in privileged mode to allow the inner container engine to manipulate the host's kernel features, such as storage drivers and networking stacks. The emergence of Podman, an Open Container Initiative (OCI) compliant tool, offers a sophisticated alternative that decouples the container engine from a central daemon, thereby facilitating a more secure and efficient execution environment. When integrated into GitLab CI, Podman enables developers to build, manage, and run containers without the inherent risks of privileged escalation or the overhead of a persistent background service. This integration is particularly critical in modern Kubernetes and OpenShift environments where strict security policies often prohibit the use of privileged containers. By leveraging Podman's ability to operate in a rootless capacity, organizations can satisfy stringent compliance requirements while maintaining the high-velocity build cycles required for modern software delivery.

The Architectural Shift from Docker-in-Docker to Podman

The primary driver for adopting Podman within GitLab CI environments is the elimination of the Docker-in-Docker (DinD) requirement. In a traditional DinD setup, a GitLab Runner must spawn a container that possesses elevated privileges to manage its own internal container engine. This creates a "privileged container" which, if compromised, could potentially allow an attacker to gain control over the underlying host or the Kubernetes node.

Podman fundamentally alters this threat model by operating as a daemonless engine. Because Podman does not require a central process running with root authority to manage container lifecycles, it can execute container operations as standard user processes. This transition has several immediate impacts on pipeline architecture:

• Elimination of the Docker daemon sidecar service.
• Reduction of the attack surface by removing the need for --privileged flags in many configurations.
• Simplification of the Kubernetes pod manifest, as there is no longer a requirement to manage a secondary daemon container alongside the build container.
• Improved resource efficiency due to the lack of a background daemon consuming memory and CPU cycles during idle periods.

However, the transition is not purely a matter of swapping commands. Whether a runner can completely avoid privileged mode depends heavily on the specific executor configuration and the underlying cluster environment. While Podman can run without a daemon, certain kernel-level features required for complex container layering or networking may still require specific configurations or security contexts depending on the host's capability.

Configuring Podman for Kubernetes and OpenShift Environments

Deploying Podman within a Kubernetes-based GitLab Runner requires a nuanced understanding of how container engines interact with the host's kernel and the container orchestrator's security constraints. In non-OpenShift Kubernetes clusters, users often find that they can run Podman as a non-root user, but specific configurations are required to ensure that the container engine can successfully perform image builds and filesystem operations.

When running Podman on a non-OpenShift Kubernetes cluster, a common method for ensuring functionality is to allow the runner to execute with the --privileged flag set to true. While this reintroduces some of the security concerns associated with DinD, it provides the container engine with the necessary access to launch containers with various security controls. For organizations that require a strict non-root user approach with non-root container processes, a specific configuration pattern must be followed in the .gitlab-ci.yml file.

To implement a non-root Podman build within a GitLab CI job, the following configuration structure is recommended:

```yaml
variables:
HOME: /mycustomdir
DOCKER_HOST: tcp://docker:2375

podman-privileged-test:
image: quay.io/podman/stable
before_script:
- podman info - id
script:
- podman build . -t playground-bis:testing
```

In this specific configuration, the HOME variable is redirected to a custom directory to avoid permission conflicts in the default user directory, and the DOCKER_HOST is defined to point to the appropriate socket, even though Podman is daemonless. This setup allows the Podman environment to simulate the expected interaction patterns of traditional container engines while maintaining the security benefits of the Podman binary itself.

Implementing the Shell Executor for High-Performance Podman Builds

While the container executor is the most common way to run GitLab CI jobs, the shell executor offers a unique advantage for Podman users. By using a shell executor on a dedicated machine where Podman is natively installed, the overhead of "nesting" containers (running a container inside a container) is entirely removed. This results in faster execution times and a more direct interface with the host's hardware and filesystem.

The following example demonstrates how a .gitlab-ci.yml file would look when utilizing a shell executor with specific tags to target a machine equipped with Podman:

yaml build-shell: stage: build tags: - podman script: - podman build -t myapp:$CI_COMMIT_SHA . - podman run --rm myapp:$CI_COMMIT_SHA npm test

In this workflow, the podman tag ensures that the GitLab Runner selects a host where Podman is already present in the system PATH. The commands podman build and podman run are executed directly against the host's Podman installation. This eliminates the complexity of managing container-in-container storage drivers, such as the issues often encountered when attempting to use overlayfs inside a containerized environment where nesting is not natively supported.

Advanced Caching Strategies for Accelerated Container Builds

One of the most significant bottlenecks in CI/CD pipelines is the time spent downloading dependencies and rebuilding unchanged layers. Podman provides several mechanisms to optimize this, which can be combined with GitLab CI's native caching features to create a high-performance build environment. There are two primary layers of caching that should be implemented to maximize efficiency: the GitLab CI file cache and the Podman/registry-based image cache.

Optimizing the GitLab CI File Cache

GitLab CI's default caching mechanism involves zipping files and uploading them to an S3-compatible storage after a job completes, then downloading and unzipping them before the next job starts. While reliable, this process incurs significant compression and transfer overhead, particularly for large dependency sets. To mitigate this, users can define a specific temporary directory for Podman/Buildah's temporary data and include it in the GitLab cache paths.

A highly effective configuration involves defining a TMPDIR within the project directory. This ensures that Podman uses a location that GitLab CI can easily track and persist.

```yaml
stages:
- build

build:
stage: build
image: quay.io/podman/stable:latest
variables:
TMPDIR: ${CIPROJECTDIR}/.local/tmp
beforescript:
- mkdir -p ${CIPROJECTDIR}/.local/tmp
- echo "$CIREGISTRYPASSWORD" | podman login -u "$CIREGISTRYUSER" --password-stdin "$CIREGISTRY"
script:
- podman build --pull --cache-from "${CIREGISTRYIMAGE}" -t "${CIREGISTRYIMAGE}" .
- podman push "${CIREGISTRYIMAGE}"
cache:
paths:
- .local/tmp/buildah-cache-0
```

By setting TMPDIR to ${CI_PROJECT_DIR}/.local/tmp, the contents of the Podman/Buildah cache are stored within the workspace. When using a Containerfile (the Podman equivalent of a Dockerfile), developers can utilize the RUN --mount=type=cache instruction. This allows for the caching of dependency-heavy directories, such as:

• /go/pkg/mod for Go projects.
• /root/.npm for Node.js projects.
• /cargo/registry for Rust projects.

Podman automatically manages these directories under the defined TMPDIR, and by adding .local/tmp/buildah-cache-0 to the cache: paths: section of the .gitlab-ci.yml, these layers persist across different pipeline runs.

Leveraging Registry-Based Layer Caching

The second layer of caching involves utilizing the container registry itself to store and retrieve image layers. Instead of relying solely on local files, Podman can pull previously built layers from the registry to use as a cache source during the build process.

The command podman build --pull --cache-from "${CI_REGISTRY_IMAGE}" is essential here. The --pull flag ensures that the builder always checks for the latest version of the base image, while --cache-from instructs Podman to look at the existing image in the registry to find matching layers. This prevents the re-execution of steps that have not changed, drastically reducing the time spent in the build stage.

Deep Integration: The Podman-GitLab-Runner Implementation

For users seeking a more permanent and integrated solution, there are specialized implementations designed to facilitate the interaction between GitLab Runner and Podman. One such implementation is the podman-gitlab-runner project, which provides a framework for running the runner itself in a rootless, Podman-managed environment.

Installation and System Configuration

Setting up a rootless Podman environment for a GitLab Runner requires precise system-level adjustments, particularly on Fedora-based distributions or other Linux systems utilizing systemd. The following steps are critical for a successful installation:

1. Install the GitLab Runner binary according to standard distribution instructions.
2. Ensure the GitLab Runner version is 12.6 or higher, as this version is required to correctly handle the image tags exposed from the .gitlab-ci.yml file.
3. Configure sub-UIDs and sub-GIDs for the gitlab-runner user. This is a prerequisite for rootless container operations, as it allows the user to map internal container user IDs to a range of unprivileged IDs on the host. This is done by adding entries to /etc/subuid and /etc/subgid.
4. Enable user lingering. By default, user services are terminated when a user logs out. To ensure the gitlab-runner service remains active, execute:
   bash sudo loginctl enable-linger gitlab-runner
5. Manage SELinux contexts. If SELinux is enabled, you may encounter permission denials when the runner attempts to execute binaries. This can be mitigated by labeling the binary with the appropriate type:
   bash sudo chcon -t bin_t /usr/bin/gitlab-runner

Rootless Service Deployment

Once the system prerequisites are met, the GitLab Runner can be configured to run as a non-root service. This is achieved by adding a systemd drop-in file at /etc/systemd/system/gitlab-runner.service.d/rootless.conf with the following content:

ini [Service] User=gitlab-runner Group=gitlab-runner

After configuring the service, the user must perform a migration to ensure the cgroups behavior is correctly aligned with Podman's requirements. This silences potential warnings during job execution and ensures stable resource management:

bash sudo -iu gitlab-runner podman system migrate

Finally, the runner is registered with the GitLab instance. The process involves cloning the specific runner implementation repository and running the registration command:

bash sudo -u gitlab-runner gitlab-runner register \ --url <YOUR_GITLAB_URL> \ --registration-token <YOUR_TOKEN>

Critical Considerations for Container Nesting and Storage Drivers

A significant technical hurdle in containerized CI environments is the "nesting" problem. When attempting to run Podman inside a container (as a GitLab Runner executor), the inner containerized process often fails to access the overlayfs driver. This is because overlayfs requires specific kernel capabilities and filesystem access that are restricted by default in most container runtimes.

RedHat provides extensive documentation on how to navigate these complexities, specifically addressing both rootful and rootless scenarios for running Podman inside containers. The failure of nesting is almost always tied to the inability of the inner container to mount filesystems. To resolve this, one must either:

• Use a host-level executor (the Shell Executor) to avoid nesting entirely.
• Configure the GitLab Runner to use a storage driver that is compatible with the host's environment, though this is often complex and limited in multi-tenant Kubernetes environments.
• Ensure that the container engine has access to the necessary kernel modules via privileged mode, which should be treated as a last resort due to security implications.

Technical Comparison: Docker-in-Docker vs. Podman in GitLab CI

The following table summarizes the key technical differences between the traditional DinD approach and the Podman-based approach.

Analytical Conclusion

The transition from Docker-based CI/CD workflows to Podman-integrated pipelines represents more than a simple change in tooling; it is a strategic move toward more resilient, secure, and efficient infrastructure. By eliminating the reliance on the Docker daemon, organizations can significantly reduce the security risks associated with privileged container execution, a necessity in the era of Zero Trust security and hardened Kubernetes clusters.

The integration of Podman into GitLab CI demands a deeper understanding of Linux user namespaces, systemd user services, and advanced caching mechanisms. While the initial configuration complexity—such as managing /etc/subuid, enabling lingering, and configuring TMPDIR for Buildah—is higher than the "plug-and-play" nature of DinD, the long-term benefits are undeniable. The ability to implement a multi-layered caching strategy, combining GitLab's file cache with Podman's registry-based layer caching, provides a pathway to sub-minute build times that were previously difficult to achieve in secure, non-privileged environments.

Furthermore, the architectural flexibility offered by Podman's daemonless nature allows for a more granular approach to resource management. Pipelines become more predictable, and the elimination of the sidecar daemon reduces the "noisy neighbor" effect in shared Kubernetes environments. As container technology continues to evolve toward more decentralized and secure models, the adoption of Podman within GitLab CI becomes not just an option, but a technical imperative for engineering teams prioritizing security, speed, and architectural integrity.

Sources

1. GitLab Documentation: Use Podman with GitLab Runner on Kubernetes
2. OneUptime: Use Podman in GitLab CI
3. Dev.to: Podman on GitLab CI - Fast, Efficient Container Builds
4. GitHub: jonasbb/podman-gitlab-runner