The integration of Podman into a GitLab CI/CD ecosystem represents a significant architectural shift for DevOps engineers seeking to move away from the security implications of the Docker daemon. GitLab Runner serves as the fundamental agent responsible for executing the various stages of a pipeline—such as build, test, and deploy—as defined within the .gitlab-ci.yml configuration file. By deploying this agent within a Podman container, organizations can achieve a highly isolated, lightweight, and portable execution environment. This approach ensures that the CI/CD runner is decoupled from the host operating system, thereby reducing the surface area for potential security breaches and simplifying the management of runner instances across diverse environments, ranging from local development workstations to massive Kubernetes clusters.
The transition to Podman is primarily driven by the need for a daemonless and rootless container engine. Unlike traditional container runtimes that require a high-privilege background process, Podman operates by directly interacting with the Linux kernel via user namespaces. When paired with GitLab Runner, this capability allows for the construction of "Podman-in-Podman" (pipglr) architectures. These setups enable the execution of containerized jobs within a rootless environment, providing a level of security where even if a CI job is compromised, the attacker remains trapped within a non-privileged user context on the host machine. This deep isolation is critical for modern DevSecOps workflows where untrusted code is frequently processed.
Architectural Fundamentals of Podman-Driven GitLab Runners
At its core, the Podman-based GitLab Runner functions as an executor that manages the lifecycle of job containers. The architecture can be categorized into several operational modes: the Shell executor, the Docker executor (repurposed for Podman), and the Custom executor. Each mode presents different trade-offs regarding isolation, performance, and complexity.
The Shell executor is the simplest form of implementation, where the runner executes commands directly within the environment of the runner container itself. This is often used for lightweight tasks or when the runner container is pre-configured with all necessary dependencies. However, it lacks the granular isolation provided by spawning separate containers for every single job.
The Docker executor, when configured to point to a Podman socket, allows the GitLab Runner to utilize Podman as the underlying engine to spawn distinct, ephemeral containers for each pipeline job. This mimics the traditional Docker-based workflow but leverages Podman’s security advantages. To achieve this, the config.toml file of the GitLab Runner must be explicitly instructed to communicate with the Podman Unix socket.
| Feature | Shell Executor | Podman/Docker Executor | Custom Executor |
|---|---|---|---|
| Isolation Level | Low (Jobs run in Runner container) | High (Each job gets a new container) | Maximum (User-defined logic) |
| Complexity | Minimal | Moderate | High |
| Security Profile | Dependent on Runner container security | High (Rootless/Daemonless) | Extremely High (Customized) |
| Resource Overhead | Very Low | Moderate | Variable |
Deployment and Container Lifecycle Management
Successfully deploying a GitLab Runner via Podman requires a disciplined approach to image management and volume persistence. To begin the deployment, the official GitLab Runner image must be retrieved from a container registry.
The following sequence outlines the initial setup of the runner container:
- Pull the official image using the command
podman pull docker.io/gitlab/gitlab-runner:latest. - Verify the presence of the image in the local storage using
podman images | grep gitlab-runner. - Create a dedicated Podman volume to ensure that the runner configuration survives container restarts or deletions. This is accomplished via
podman volume create gitlab-runner-config. - Launch the container in detached mode, ensuring that the volume is correctly mounted. The command
podman run -d --name my-gitlab-runner -v gitlab-runner-config:/etc/gitlab-runner:Z docker.io/gitlab/gitlab-runner:latestutilizes the:Zflag, which is critical for SELinux-enabled systems to manage file permissions correctly. - Monitor the operational status of the container using
podman ps. - Inspect the internal logs for any initialization errors using
podman logs my-gitlab-runner.
Once the container is active, the runner must be registered with the GitLab instance. This process links the containerized agent to a specific GitLab project or instance via an authentication token. For a non-interactive registration, the following command structure is utilized:
bash
podman exec my-gitlab-runner gitlab-runner register \
--non-interactive \
--url "https://gitlab.com/" \
--token "YOUR_RUNNER_AUTHENTICATION_TOKEN" \
--executor "shell" \
--description "podman-runner"
After registration, the validity of the agent can be confirmed by querying the list of active runners from within the container:
bash
podman exec my-gitlab-runner gitlab-runner list
Advanced Configuration for Podman Socket Integration
To move beyond the Shell executor and enable the runner to spawn Podman containers for jobs, the config.toml file requires specific modifications. This is particularly important when the goal is to allow the runner to build and run images during the CI process. One of the most critical requirements for this setup is the configuration of a network per build, which ensures that each job has its own isolated network namespace.
The configuration process involves two primary steps: enabling the required feature flag and pointing the runner to the Podman socket.
- Modify the
/etc/gitlab-runner/config.tomlfile to include theFF_NETWORK_PER_BUILDfeature flag within the[[runners]]section. - Configure the
[runners.docker]block to use the Podman Unix socket instead of the default Docker socket.
An example of a highly specialized configuration for this purpose is provided below:
```toml
[[runners]]
environment = ["FFNETWORKPER_BUILD=1"]
[runners.docker]
host = "unix:///run/user/1001/podman/podman.sock"
```
The path to the socket (/run/user/1001/podman/podman.sock) is highly dependent on the UID of the user running the Podman service. Once these changes are applied, the runner must be restarted to reload the configuration:
bash
sudo gitlab-runner restart
Implementation on Kubernetes and OpenShift
When scaling GitLab Runners to a Kubernetes or OpenShift environment, the complexity of managing Podman increases. In these orchestrated environments, Podman can be used to allow users to build container images within a CI job without requiring root privileges or elevated host permissions. This is a vital capability for multi-tenant clusters where security policies strictly forbid privileged escalation.
On non-OpenShift Kubernetes clusters, users can run Podman as a non-root user by setting the --privileged flag to true. While this provides the container engine with the necessary capabilities to manage namespaces and filesystems, it must be handled with caution to maintain the security posture of the cluster.
For CI jobs that require building images, the .gitlab-ci.yml file can be configured to use a Podman-based image. The following configuration snippet demonstrates how to execute a podman build command within a CI job using the quay.io/podman/stable image:
```yaml
variables:
HOME: /mycustomdir
DOCKER_HOST: tcp://docker:2375
podman-privileged-test:
image: quay.io/podman/stable
before_script:
- podman info - id
script:
- podman build . -t playground-bis:testing
```
This configuration allows for a "rootless-in-container" workflow. The DOCKER_HOST variable is directed toward a TCP endpoint, which is a common pattern in environments where a service-based container engine is available.
Custom Executor and the podman-gitlab-runner Framework
For organizations requiring extreme customization, the use of a Custom Executor via specialized scripts offers the highest degree of control. The podman-gitlab-runner project provides a framework for this by utilizing prepare, run, and cleanup scripts. This method allows an engineer to redefine exactly how Podman spawns containers and how dependencies are injected into the runtime environment.
To implement this custom framework, the runner is registered with specific execution hooks:
bash
sudo -u gitlab-runner gitlab-runner register \
--url https://my.gitlab.instance/ \
--registration-token $GITLAB_REGISTRATION_TOKEN \
--name "Podman Runner" \
--executor custom \
--builds-dir /home/user \
--cache-dir /home/user/cache \
--custom-prepare-exec "/home/gitlab-runner/podman-gitlab-runner/prepare.sh" \
--custom-run-exec "/home/gitlab-runner/podman-gitlab-runner/run.sh" \
--custom-cleanup-exec "/home/gitlab-runner/podman-gitlab-runner/cleanup.sh"
The framework allows for the overriding of default behaviors through environment variables and file renaming. By renaming the template file custom_base.template.sh to custom_base.sh, users can access deeper customization layers.
Key supported variables for customization include:
PODMAN_RUN_ARGS: This variable allows the user to pass specific flags to the Podman command when it spawns job containers, such as volume mounts, security labels, or resource limits.
Furthermore, managing access to private registries is a common requirement in enterprise environments. This is handled by setting the DOCKER_AUTH_CONFIG variable within the GitLab CI/CD settings. This variable provides the necessary credentials (usually in a JSON format) that Podman uses to authenticate against private registries during the podman build or podman pull phases.
Scalability and High Availability Strategies
In high-demand environments, a single GitLab Runner instance may become a bottleneck. To achieve parallel job execution and increase throughput, multiple runner instances should be deployed. This is achieved by creating separate volumes for each runner to prevent configuration collisions and state corruption.
The following workflow demonstrates the creation of a multi-runner architecture:
Create distinct volumes for each runner's configuration:
podman volume create runner-1-configpodman volume create runner-2-config
Launch separate runner containers, each mapped to its own volume:
podman run -d --name gitlab-runner-1 -v runner-1-config:/etc/gitlab-runner:Z docker.io/gitlab/gitlab-runner:latestpodman run -d --name gitlab-runner-2 -v runner-2-config:/etc/gitlab-runner:Z docker.io/gitlab/gitlab-runner:latest
Register each runner independently using their respective tokens. This allows for granular control, such as assigning different runner tags to different containers (e.g., one runner specifically for "build" jobs and another for "test" jobs).
To verify the registration of these multiple runners, the list command can be executed within each container to ensure they are distinct and correctly identified by the GitLab instance.
Troubleshooting and Operational Stability
Operating a GitLab Runner via Podman is generally stable, but certain edge cases can lead to pipeline failures. One reported issue involves runners crashing after a specific number of builds (e.g., every 100 builds). Such failures are often indicative of resource exhaustion or state accumulation.
Common causes for runner instability include:
- Disk space depletion caused by unmanaged container images and build caches.
- Socket exhaustion if the Podman service is not recycling connections correctly.
- Memory leaks within the runner process or the underlying Podman engine.
- SELinux denials that prevent the runner from accessing necessary host resources.
To mitigate these issues, it is essential to implement regular cleanup routines. This can be done by periodically running podman system prune or by configuring the GitLab Runner to use a dedicated, clean environment for every job via the FF_NETWORK_PER_BUILD flag and ensuring that the cleanup.sh script in a custom executor is robust enough to remove ephemeral artifacts.
Analysis of the Podman-GitLab Integration
The deployment of Podman as a GitLab Runner executor is more than a simple change in tooling; it is a strategic decision that impacts the security architecture of the entire CI/CD pipeline. By moving to a daemonless and rootless model, organizations significantly reduce the risk of privilege escalation attacks, which is a primary concern in shared computing environments like Kubernetes.
The implementation complexity is higher than standard Docker-based setups, particularly regarding the configuration of Unix sockets and the management of custom executors. However, the benefits of portability and isolation are substantial. The ability to run "Podman-in-Podman" provides a clear path for teams to implement highly secure, multi-tenant CI/CD pipelines that can run on any OCI-compliant platform without requiring host-level root access.
Ultimately, the success of a Podman-based GitLab Runner implementation depends on the meticulous configuration of the config.toml file, the careful management of volumes for persistence, and the proactive monitoring of resource usage to prevent the periodic failures common in unoptimized containerized environments. As the industry continues to move toward zero-trust architectures, the adoption of rootless container runtimes like Podman for CI/CD execution will likely become a standard requirement rather than an optional optimization.