Architecting Secure Containerized Pipelines with Podman in GitLab CI

The integration of Podman into GitLab CI/CD workflows represents a fundamental shift in how DevOps engineers approach container orchestration and security within automated pipelines. For years, the industry standard for containerized builds within GitLab has relied heavily on Docker-in-Docker (DinD). While functional, DinD introduces significant architectural complexities and, more critically, severe security vulnerabilities. Because DinD requires the Docker daemon to run in privileged mode to facilitate the nesting of containers, it opens a potential attack vector where a compromised build process could escalate privileges to the host machine. Podman provides a sophisticated alternative by operating as a daemonless, Open Container Initiative (OCI) compliant tool. This architectural distinction allows for the execution of container builds without the requirement of a central daemon service, thereby simplifying the pipeline topology and dramatically hardening the security posture of the entire CI/CD ecosystem.

The transition from Docker-based pipelines to Podman-driven architectures is not merely a swap of tools; it is a reconfiguration of the trust model within the CI environment. By leveraging Podman's ability to run rootless containers, organizations can ensure that even if a build job is compromised, the impact is strictly confined to the unprivileged user namespace. This deep integration of security and efficiency allows for faster, more predictable builds that align with modern DevSecOps principles, especially when deployed across diverse environments ranging from local shell executors to complex Kubernetes clusters and OpenShift deployments.

Architectural Advantages of Daemonless Container Engines

The primary driver for adopting Podman within GitLab CI is the elimination of the Docker daemon dependency. In a traditional DinD setup, the GitLab Runner must invoke a Docker daemon, which often requires the container to be run with the --privileged flag. This flag essentially grants the container nearly all the capabilities of the host kernel, bypassing many of the isolation boundaries that containers are intended to provide.

Podman's daemonless nature changes this dynamic entirely. Because Podman does not require a background service to manage container lifecycles, it can be invoked directly as a standard process. This leads to several critical impacts:

• Reduced overhead: Without a daemon to manage, there is less background resource consumption on the runner host.
• Improved security isolation: Processes run directly under the user identity, facilitating a rootless model.
• Simplified orchestration: The lack of a sidecar daemon service reduces the moving parts within the GitLab CI YAML configuration.

Configuring GitLab Runner Executors for Podman Integration

To implement Podman within a GitLab CI environment, the configuration of the GitLab Runner executor is the most critical technical step. The approach varies significantly depending on whether the runner is utilizing a shell executor or a container-based executor.

The Shell Executor Approach

The shell executor is the most straightforward method for implementing Podman. In this scenario, the GitLab Runner is installed directly on a host machine where Podman is already present and configured. The runner executes commands directly in the host's shell environment. This is particularly effective for dedicated build servers where the overhead of containerizing the runner itself is undesirable.

A typical implementation for a shell executor involves using specific tags to ensure the job is routed to a machine equipped with Podman.

yaml build-shell: stage: build tags: - podman script: - podman build -t myapp:$CI_COMMIT_SHA . - podman run --rm myapp:$CI_COMMIT_SHA npm test

In this configuration, the tags attribute ensures that the GitLab Runner identifies the job as one that must be handled by a runner with the podman label. The script then directly calls the podman binary, which is available in the host's $PATH. This eliminates the need for any complex container nesting, as the build process is native to the host's execution environment.

The Container Executor and Kubernetes Environments

For more scalable and dynamic environments, such as Kubernetes or OpenShift, the container executor is preferred. Here, the GitLab Runner starts a new container to run the job. When using Podman within a containerized runner, the challenge is "nesting" the container engine.

On non-OpenShift Kubernetes clusters, running Podman typically requires setting the --privileged flag to true. While this provides the necessary permissions for the container engine to function, it does increase the security surface area. However, even with privileged mode enabled, Podman offers configurations that allow for building container images without requiring a root user for the actual container processes.

To run Podman as a non-root user within a Kubernetes-based GitLab Runner, specific variables must be defined in the .gitlab-ci.yml file to direct the environment behavior.

```yaml
variables:
HOME: /mycustomdir
DOCKER_HOST: tcp://docker:2375

podman-privileged-test:
image: quay.io/podman/stable
before_script:
- podman info -id
script:
- podman build . -t playground-bis:testing
```

This specific configuration uses the quay.io/podman/stable image, which is a specialized environment containing the necessary Podman binaries and dependencies. The before_script section is used to validate the environment via podman info, ensuring the runtime is correctly initialized before the build begins.

Rootless Implementation and System Configuration

Achieving a truly secure, rootless Podman setup within GitLab CI requires meticulous configuration at the OS and service levels. A rootless setup ensures that the gitlab-runner user operates within its own unprivileged user namespace, preventing any accidental or malicious interaction with host-level system files.

Essential Host-Level Preparations

If you are managing your own runners (for example, on a Fedora-based system), several system-level steps must be completed to allow the gitlab-runner user to manage containers effectively.

1. Subuid and Subgid Configuration: Podman relies on user namespaces to map internal container users to external host users. You must ensure that entries exist in /etc/subuid and /etc/subgid for the gitlab-runner user. Without these, the kernel will not allow the user to map the necessary ranges for rootless container execution.

2. Enabling User Lingering: By default, many Linux distributions terminate user processes when the user logs out. Since the gitlab-runner is a service user, you must enable "lingering" to ensure its processes (including the Podman daemonless containers) can persist and run independently of an active session.

bash sudo loginctl enable-linger gitlab-runner

3. Cgroup Management: To prevent warnings and ensure correct resource control, you should run the system migration command as the gitlab-runner user. This optimizes how cgroups are handled in the rootless environment.

bash sudo -iu gitlab-runner podman system migrate

4. SELinux Labeling: If running on a system with SELinux enabled (like Fedora or RHEL), you may encounter permission denials when the GitLab Runner attempts to execute binaries. You can resolve this by applying the appropriate bin_t type to the GitLab Runner binary.

bash sudo chcon -t bin_t /usr/bin/gitlab-runner

Systemd Service Customization

For advanced deployments, such as running the GitLab Runner as a rootless service itself, you must modify the systemd unit file. This is achieved by adding a drop-in configuration file to ensure the service runs under the gitlab-runner user and group rather than root.

Create the following directory and file: /etc/systemd/system/gitlab-runner.service.d/rootless.conf

ini [Service] User=gitlab-runner Group=gitlab-runner

This configuration ensures that the entire lifecycle of the runner—from job pickup to container execution—is contained within the unprivileged user space.

High-Performance Caching Strategies

One of the most significant bottlenecks in CI/CD pipelines is the time spent downloading dependencies and rebuilding layers. Podman, when combined with GitLab CI's caching mechanisms, offers a dual-layered approach to optimization that can drastically reduce build durations.

The Dual-Layer Caching Model

Effective Podman caching in GitLab CI utilizes two distinct but complementary layers:

• GitLab CI File Cache: This involves persisting specific directories within the project workspace across different jobs and pipeline runs.
• Registry-Based Image Cache: This involves pushing and pulling intermediate container layers from a container registry.

Optimizing Workspace Caching

To maximize the efficiency of the GitLab CI file cache, you must explicitly define where Podman stores its temporary build data. By directing Podman to use a directory within the CI project folder, you ensure that GitLab's caching mechanism can "see" and upload those files to its storage backend (such as S3).

The following steps are required for a high-performance setup:

1. Define a temporary directory in your CI variables.

yaml variables: TMPDIR: ${CI_PROJECT_DIR}/.ci/tmp

2. Ensure the directory exists before the build starts using the before_script instruction.

yaml before_script: - mkdir -p "$TMPDIR"

3. Add the directory to the cache:paths section of your .gitlab-ci.yml file.

yaml cache: paths: - .ci/tmp/

Leveraging Buildah Cache Mounts

For dependency-heavy builds (e.g., Go, Node.js, or Rust), using RUN --mount=type=cache within your Containerfile (or Dockerfile) is highly recommended. This allows the build process to mount a persistent cache directory directly into the build container. When TMPDIR is correctly mapped to the GitLab CI cache, Podman automatically handles these directories, allowing dependencies to persist across pipeline runs without being re-downloaded.

Common directories to cache include:
- /go/pkg/mod (Go modules)
- /root/.npm (NPM packages)
- /cargo/registry (Rust crates)

Implementation Example: Optimized Build Job

The following example demonstrates a complete, optimized CI job that implements both registry-based caching and workspace-based temporary directory caching.

```yaml
stages:
- build

build:
stage: build
image: quay.io/podman/stable:latest
variables:
TMPDIR: ${CIPROJECTDIR}/.local/tmp
beforescript:
- mkdir -p ${CIPROJECTDIR}/.local/tmp
- echo "$CIREGISTRYPASSWORD" | podman login -u "$CIREGISTRYUSER" --password-stdin "$CIREGISTRY"
script:
- podman build --pull --cache-from "${CIREGISTRYIMAGE}" -t "${CIREGISTRYIMAGE}" .
- podman push "${CIREGISTRYIMAGE}"
cache:
paths:
- .local/tmp/buildah-cache-0
```

In this workflow, the --cache-from flag instructs Podman to check the container registry for existing layers that match the current build, significantly reducing the amount of work the engine has to perform. Simultaneously, the cache:paths directive ensures that local Buildah data is preserved for subsequent jobs.

Advanced Considerations and Troubleshooting

While the Podman-based approach is superior in security and architecture, it requires a deeper understanding of the underlying container technology.

OverlayFS and Nesting Failures

A common issue encountered when attempting to run Podman inside a container (container-in-container) is the failure of the overlayfs mount. By default, most container runtimes restrict access to the host's overlay filesystem to prevent security breaches. This often results in errors where the container engine cannot create the necessary layer structures for a build.

As noted in technical documentation, RedHat provides specific guidance for handling both rootful and rootless scenarios when running Podman inside containers. If you encounter overlayfs errors, you may need to utilize the vfs driver as a fallback, although this is significantly slower and less storage-efficient than overlayfs.

Performance in Self-Managed GitLab Environments

In self-managed GitLab instances, users have the opportunity to further optimize caching by mounting S3 storage directly via FUSE or other network filesystems. This technique eliminates the overhead of zipping and unzipping cache archives—a process that can be extremely time-consuming for large dependency sets. When using FUSE, the cache behaves like a live directory, allowing Podman to access files instantly, which provides a dramatic speed increase for large-scale enterprise pipelines.

Technical Analysis and Conclusion

The transition from Docker-in-Docker to Podman within GitLab CI is a move toward architectural maturity. By removing the requirement for a centralized, privileged daemon, organizations solve two of the most persistent problems in CI/CD: security vulnerabilities and configuration complexity.

The implementation of Podman requires a more nuanced understanding of the host system—specifically regarding user namespaces (subuid/subgid), SELinux labeling, and systemd service management. However, the rewards of this complexity are a significantly hardened build environment where the "blast radius" of a compromised job is limited by the unprivileged user namespace.

Furthermore, the sophisticated caching strategies available through the combination of GitLab's file cache, Podman's registry-based layer caching, and Buildah's mountable cache directories allow for performance levels that can match or exceed traditional Docker setups. As DevSecOps continues to prioritize "security by design," the adoption of daemonless, rootless container engines like Podman is no longer just an option; it is a necessity for professional-grade automated software delivery.

Sources

1. OneUptime: Podman with GitLab CI
2. GitLab Documentation: Use Podman with Kubernetes
3. Dev.to: Podman on GitLab CI - Fast and Efficient
4. GitHub: jonasbb/podman-gitlab-runner