The integration of GitLab Runner with Podman represents a significant shift in the landscape of Continuous Integration and Continuous Deployment (CI/CD) architectures. For years, the industry standard for running automated build and test jobs has relied heavily on the Docker daemon. However, as security paradigms shift toward zero-trust models and rootless execution, Podman has emerged as a formidable, daemonless, and security-centric alternative. By utilizing Podman as the underlying engine for a GitLab Runner, organizations can achieve a highly isolated, portable, and lightweight execution environment that mitigates many of the security risks inherent in traditional root-privileged container runtimes. This technological convergence allows for the automation of application compilation, testing, and deployment within a framework that prioritizes the principle of least privilege.
A GitLab Runner functions as a specialized agent designed to interface with GitLab's CI/CD infrastructure. It interprets the instructions defined within a project's .gitlab-ci.yml file and executes the corresponding jobs. While GitLab provides shared runners for its public instance, many enterprise-level users opt to deploy their own dedicated runners. This decision is often driven by the need for granular control over the underlying infrastructure, enhanced privacy for proprietary codebases, more flexible configuration options, or the necessity to manage specific resource allocations that exceed the limits of standard shared runner accounts. When these runners are deployed using Podman, the runner itself can be encapsulated within a container, decoupling the CI/CD logic from the host operating system and facilitating seamless migration across various environments, including local development machines, on-premises servers, and complex Kubernetes clusters.
Architectural Foundations of Podman and GitLab Runner Integration
To understand the implementation of this stack, one must first distinguish between the roles of the executor and the container engine. The GitLab Runner acts as the orchestrator, but it requires an executor to handle the actual lifecycle of the job container. Historically, the Docker executor has been the default choice. Because Podman is an Open Container Initiative (OCI) compliant tool, it is designed to be a drop-in replacement for Docker in many scenarios. This compatibility is achieved through Podman's ability to provide a Docker-compatible API socket.
The shift from Docker to Podman introduces several critical architectural benefits and requirements:
- Lightweight execution: Podman operates without a persistent background daemon, reducing the resource footprint on the host system.
- Enhanced Isolation: By running the runner in a Podman container, the execution environment is strictly isolated from the host system, which minimizes the blast radius of a potential security breach within a pipeline job.
- Portability: Containerizing the GitLab Runner via Podman ensures that the runner's configuration and dependencies are packaged together, allowing for consistent behavior across different host distributions.
- Rootless Capabilities: Podman allows for the execution of containers without requiring root privileges on the host, a feature that is instrumental in securing CI/CD pipelines against privilege escalation attacks.
| Feature | Docker Executor | Podman (via Socket) |
|---|---|---|
| Daemon Requirement | Mandatory central daemon | Daemonless (uses per-user sockets) |
| Security Model | Traditionally root-privileged | Supports native rootless execution |
| API Compatibility | Standard Docker API | Docker-compatible via podman.socket |
| Resource Overhead | Higher due to daemon management | Lower due to direct process execution |
Implementing Podman on Non-OpenShift Kubernetes Clusters
In cloud-native environments, specifically within Kubernetes, the deployment of Podman requires precise configuration to ensure that container images can be built and managed during a CI job without violating cluster security policies. On non-OpenShift Kubernetes clusters, the configuration must account for the user's privilege level.
To execute Podman as a non-root user while maintaining the ability to perform necessary container operations, the container must be launched with the --privileged flag set to true. This flag instructs the container engine to launch the container with expanded security permissions, bypassing certain host-level security controls that would otherwise prevent container-in-container operations.
When configuring a pipeline to build images using Podman within a Kubernetes-based GitLab Runner, the .gitlab-ci.yml file must be structured to provide the necessary environment variables and execution steps. For example, a job designed to test Podman's functionality in a non-root environment would look like this:
```yaml
variables:
HOME: /mycustomdir
DOCKER_HOST: tcp://docker:2375
podman-privileged-test:
image: quay.io/podman/stable
before_script:
- podman info - id
script:
- podman build . -t playground-bis:testing
```
In this configuration, the DOCKER_HOST variable is redirected to a TCP address, and the HOME variable is redirected to a custom directory to ensure that the non-root user has a valid, writable location for configuration and storage. This setup allows the CI job to leverage Podman's capabilities within the constraints of the Kubernetes pod.
Configuring the User-Scoped Podman Socket for GitLab Runner
Because Podman is architected to run separately for each Linux user, it does not rely on a single, global system daemon like Docker. Instead, it utilizes a per-user socket. For a GitLab Runner to communicate with Podman, the runner must be able to access a specific Unix socket that serves as the Docker-compatible interface.
The setup process for a dedicated gitlab-runner user involves enabling user-scoped systemd services to ensure the socket is available and persistent. The following sequence of commands outlines the technical procedure for preparing the user environment and activating the socket:
Switch to the target user account:
sudo su -l gitlab-runnerEnable user lingering to allow systemd services to run even when the user is not logged in:
loginctl enable-linger gitlab-runnerConfigure the XDG runtime directory to ensure the user-scoped environment is correctly initialized:
echo 'export XDG_RUNTIME_DIR=/run/user/$(id -u)' >> .bash_profile
source .bash_profileEnable and start the Podman socket service for the current user:
systemctl --user enable --now podman.socketRetrieve the specific path to the active socket:
systemctl --user status podman.socket | grep Listen
The output of the final command will provide a path similar to /run/user/978/podman/podman.sock. This path is critical for the subsequent GitLab Runner configuration.
Once the socket path is identified, it must be manually injected into the GitLab Runner's config.toml file. The runner is not natively aware of the Podman socket location and must be explicitly instructed to look there instead of the standard Docker socket path. Furthermore, to ensure proper networking isolation between different jobs, the FF_NETWORK_PER_BUILD feature flag should be enabled.
The modified [[runners]] section of the config.toml should be configured as follows:
toml
[[runners]]
environment = ["FF_NETWORK_PER_BUILD=1"]
[runners.docker]
host = "unix:///run/user/978/podman/podman.sock"
After updating the configuration, the runner must be restarted to apply the changes:
bash
sudo gitlab-runner restart
Following the restart, the administrator must verify the runner's status within the GitLab project's Settings -> CI/CD -> Runners interface. Successful integration is also confirmed by inspecting the CI task logs, which will display the runner's name and the specific feature flags and container images being utilized during the execution.
Advanced Customization via the Custom Executor Method
For users requiring even deeper integration or specific behaviors that the standard Docker-compatible mode cannot provide, the GitLab Runner's "custom executor" can be utilized. This method involves using scripts to manage the lifecycle of the build environment. A notable implementation of this is found in the podman-gitlab-runner project, which provides a framework for running rootless Podman via custom execution scripts.
To implement this approach, the gitlab-runner user must clone the necessary repository and register the runner with specific parameters that point to custom preparation, run, and cleanup scripts.
The registration command follows this structure:
bash
sudo -u gitlab-runner gitlab-runner register \
--url https://my.gitlab.instance/ \
--registration-token $GITLAB_REGISTRATION_TOKEN \
--name "Podman Runner" \
--executor custom \
--builds-dir /home/gitlab-runner/builds \
--cache-dir /home/gitlab-runner/cache \
--custom-prepare-exec "/home/gitlab-runner/podman-gitlab-runner/prepare.sh" \
--custom-run-exec "/home/gitlab-runner/podman-gitlab-runner/run.sh" \
--custom-cleanup-exec "/home/gitlab-runner/podman-gitlab-runner/cleanup.sh"
In this advanced configuration, the prepare.sh, run.sh, and cleanup.sh scripts handle the heavy lifting of container management. While the baseline scripts may offer limited functionality, they can be extensively customized by modifying the start_container and install_dependencies functions.
To enable high-level customization, users should rename the custom_base.template.sh file to custom_base.sh. This allows for the injection of specific environment variables that dictate how Podman behaves. Key variables include:
PODMAN_RUN_ARGS: This variable allows the user to pass specific arguments to the Podman command, such as volume mounts, security opts, or resource limits, thereby customizing how containers are spawned.
Furthermore, Podman's ability to access private registries must be addressed. This is handled by defining the DOCKER_AUTH_CONFIG variable within the GitLab project's Settings -> CI / CD settings. This variable contains the necessary credentials to authenticate against private container registries, ensuring that the Podman-powered runner can pull required images securely.
Troubleshooting and Stability Considerations
While the transition to Podman provides significant security advantages, it is not without operational challenges. One reported issue in the community involves the stability of GitLab Runners running on Linux distributions like Debian via Podman. Specifically, some users have observed instances where the runner crashes after a specific number of builds (e.g., every 100 builds). Such failures can lead to persistent pipeline failures and require deep investigation into resource exhaustion, socket timeouts, or potential leaks in the user-scoped systemd services.
When troubleshooting these stability issues, administrators should focus on several diagnostic areas:
- Systemd Journal Logs: Inspecting
journalctl --user -u podman.socketcan reveal if the socket service is failing or restarting unexpectedly. - Resource Monitoring: Check if the user-scoped processes are hitting cgroup limits or memory constraints.
- Socket Persistence: Verify if the
XDG_RUNTIME_DIRremains consistent and that the socket file does not become stale or corrupted during high-frequency job execution. - Feature Flag Interactions: Ensure that
FF_NETWORK_PER_BUILD=1is not creating a conflict with Podman's internal networking driver in high-concurrency scenarios.
Technical Summary of Podman-GitLab Integration
The implementation of Podman as a GitLab Runner executor is a multi-layered process that requires a deep understanding of both the GitLab Runner configuration and the Linux user-space management system.
| Implementation Layer | Primary Requirement | Key Configuration Detail |
|---|---|---|
| User Environment | User Lingering | loginctl enable-linger $USER |
| Service Layer | User-scoped Systemd | systemctl --user enable --now podman.socket |
| Runner Configuration | Socket Redirection | host = "unix:///run/user/[UID]/podman/podman.sock" |
| Network Layer | Build Isolation | FF_NETWORK_PER_BUILD=1 |
| Advanced Orchestration | Custom Executor | prepare.sh, run.sh, cleanup.sh |
The deployment of Podman-powered runners represents a sophisticated approach to CI/CD, moving away from the monolithic, root-dependent daemon model toward a modular, user-centric, and highly secure architecture. By leveraging the Podman socket and custom executor capabilities, DevOps engineers can build pipelines that are not only automated but are also fundamentally more resilient and aligned with modern security principles.