Decoupling GitLab CI/CD from the Docker Daemon via Podman and Optimized Execution

The architectural reliance on Docker within GitLab CI/CD pipelines has long been a standard, yet it introduces significant security vectors and performance bottlenecks. For organizations deploying services to Kubernetes environments, such as AWS EKS, the traditional approach involves utilizing Docker-in-Docker (DinD). This method allows the execution of docker commands within a pipeline script to build and publish images. However, the requirement for privileged mode to run the Docker daemon creates a critical vulnerability; because Docker daemons operate with root privileges, they become primary targets for attackers seeking to escalate privileges from a container to the host machine. Transitioning away from this privileged model toward tools like Podman, or optimizing the execution environment to minimize daemon dependency, is essential for maintaining a hardened security posture in modern DevOps infrastructures.

The Architectural Burden of Docker-in-Docker

In standard GitLab CI/CD configurations, the docker:dind service is frequently invoked to enable the building of container images. This is typically seen in the .gitlab-ci.yml file where the services keyword is used to spin up a separate container running the Docker daemon.

The operational flow involves a builder image, such as a customized Java builder based on ibm-semeru-runtimes:open-17.0.5_8-jdk-jammy, which has the Docker tool installed via a shell script:

bash curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh

When a job is executed, the GitLab Runner manages the lifecycle of the pod. If the runner is hosted on a Kubernetes cluster for tooling purposes, each job is instantiated as its own pod. The inclusion of the docker:19.03.12-dind service ensures that the docker build and docker push commands in scripts like build.sh or publish.sh have a daemon to communicate with.

The impact of this architecture is a "privileged" requirement. Without privileged mode, the inner Docker daemon cannot manage the host's network or storage drivers, rendering the build process impossible. This creates a conflict between the need for agility in building images and the security mandate to operate with the least privilege possible.

Transitioning to Podman for Daemonless Builds

To eliminate the security risks associated with privileged Docker-in-Docker, Podman emerges as a viable alternative. Podman provides a pod-manager that does not require a background daemon to function, effectively removing the need for the privileged mode that is the hallmark of DinD.

By utilizing Podman, the pipeline can execute container build and push operations without granting the container root-level access to the host kernel. This shift transforms the security profile of the CI/CD pipeline from a potential attack vector into a secure, isolated process.

The practical implementation involves replacing the services: - docker:dind declaration with a toolset that supports rootless container management. This ensures that even if a job is compromised, the attacker does not possess root access to the underlying Kubernetes node in the tooling cluster.

Pipeline Anatomy and Execution Phases

Understanding how to move away from Docker requires a deep understanding of the GitLab job lifecycle. Every job undergoes a specific set of phases that impact performance and resource utilization.

1. Pending State: The job is created and waits for an available runner.
2. Execution Environment Preparation: The runner identifies the specified image in .gitlab-ci.yml and pulls it.
3. Container Instantiation: A container is created from the pulled image.
4. Repository Cloning: The runner clones the git repository into the container.
5. Script Execution: The commands defined in the script section are executed against the codebase.
6. Artifact and Cache Management: The job pulls existing caches and pushes new artifacts or caches to object storage (such as S3 or MinIO).

When removing Docker from the equation, the "Preparation" and "Instantiation" phases become critical. Using versioned public CI Docker images is a best practice to ensure consistency and avoid the "latest" tag volatility, which can lead to unpredictable pipeline failures.

Advanced Pipeline Optimization Strategies

Reducing pipeline execution time is often as important as removing the Docker daemon. In professional React project deployments, pipelines can be reduced from 14 minutes to under 3 minutes through iterative optimization.

Caching and Artifact Management

The misuse of caches and artifacts is a common anti-pattern. Caching should be used for dependencies that are downloaded from the internet, while artifacts should be used for files produced by the job that are needed by subsequent stages.

In a Node.js environment, installing node_modules repeatedly is a costly operation. To optimize this, the node_modules folder should be cached using the yarn.lock file as the cache key.

• Impact of Caching: By caching dependencies, a pipeline can be accelerated by more than 2 minutes.
• Cache Invalidation: Using yarn.lock ensures that the cache is only rebuilt when dependencies actually change, preventing the use of stale packages.
• Storage Backends: GitLab can store these files in the container filesystem or external object storage like S3 or MinIO.

Intelligent Job Orchestration

To move beyond linear execution and further decrease the reliance on heavy container restarts, developers should implement the following logic:

The Rules Keyword

GitLab recommends the rules keyword over the deprecated only and except keywords. This allows for complex conditional logic to determine when a job should run. For example:

• Build and Docker Build jobs should only run on the main branch.
• Test jobs should only run on merge requests.

This reduces the number of unnecessary containers instantiated, thereby reducing the overall load on the runner infrastructure.

DAG (Directed Acyclic Graph) Pipelines

Standard pipelines execute jobs in stages; no job in a stage begins until all jobs in the previous stage complete. This is inefficient for large multi-tier projects or mono-repos. DAG pipelines allow jobs to start as soon as their specific dependencies are met, regardless of the stage. This is achieved using the needs keyword.

Implementation of a Continuous Deployment Workflow

A complete transition to a production-ready pipeline involves several distinct stages: publish, staging, release, version, and production. Even when moving away from the Docker daemon internally for builds, the final output is often a container image that must be pushed to a registry.

The following configuration demonstrates a structured approach to these stages:

```yaml
variables:
TAGLATEST: $CIREGISTRYIMAGE/$CICOMMITREFNAME:latest
TAGCOMMIT: $CIREGISTRYIMAGE/$CICOMMITREFNAME:$CICOMMITSHA
STAGINGTARGET: $STAGINGTARGET
PRODUCTIONTARGET: $PRODUCTIONTARGET

stages:
- publish
- staging
- release
- version
- production

publish:
stage: publish
image: docker:latest
services:
- docker:dind
rules:
- if: $CICOMMITBRANCH == "main" && $CICOMMITTAG == null
script:
- docker build -t $TAGLATEST -t $TAGCOMMIT .
- docker login -u $CIREGISTRYUSER -p $CIREGISTRYPASSWORD $CIREGISTRY
- docker push $TAGLATEST
- docker push $TAG_COMMIT

staging:
stage: staging
image: alpine:latest
rules:
- if: $CICOMMITBRANCH == "main" && $CICOMMITTAG == null
script:
- chmod 400 $GITLABKEY
- apk add openssh-client
- docker login -u $CIREGISTRYUSER -p $CIREGISTRYPASSWORD $CIREGISTRY
- ssh -i $GITLABKEY -o StrictHostKeyChecking=no $USER@$STAGINGTARGET "
docker pull $TAGCOMMIT &&
docker rm -f myapp || true &&
docker run -d -p 80:80 --name myapp $TAGCOMMIT"
environment:
name: staging
url: http://$STAGING_TARGET

releasejob:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
rules:
- if: $CICOMMITTAG
script:
- |
DEPLOYTIME=$(date '+%Y-%m-%d %H:%M:%S')
CHANGES=$(git log $(git describe --tags --abbrev=0))
```

Comparison of Container Execution Methods

The following table delineates the differences between traditional Docker-in-Docker and the proposed daemonless/optimized approaches.

Best Practices for Avoiding Anti-Patterns

To ensure the long-term maintainability of GitLab CI/CD without falling into common traps, the following strategies should be adopted:

• Image Strategy: Start with versioned public CI Docker images to avoid dependency drift.
• Repository Structure: Utilize mono-repos for new products but manage them with DAG pipelines to avoid bottlenecks.
• Configuration Management: Avoid local GitLab CI YAML files; use centralized configurations and abstract duplicated code without relying on YAML anchors.
• Scripting: Avoid raw commands in scripts; wrap them in maintainable shell scripts.
• Pipeline Structure: Split jobs wisely to ensure failure in one component does not block the entire pipeline.
• Downstream Pipelines: Avoid overusing downstream pipelines, as they can obscure visibility and complicate the dependency graph.

Conclusion

The migration away from privileged Docker-in-Docker in GitLab CI/CD is not merely a technical preference but a security imperative. By integrating Podman and adopting rootless container strategies, organizations can mitigate the risks associated with root-privileged daemons. Furthermore, the transition to a more efficient pipeline involves moving beyond simple stage-based execution toward Directed Acyclic Graphs (DAG), utilizing the rules keyword for precise job triggering, and implementing aggressive caching strategies for dependency management. The synergy of these optimizations—reducing a pipeline from 14 minutes to 3 minutes—demonstrates that the removal of the Docker daemon is part of a broader evolution toward leaner, faster, and more secure software delivery lifecycles. The ultimate goal is a pipeline that provides rapid feedback to developers without compromising the integrity of the underlying infrastructure.

Sources

1. Trifork Blog
2. Theodo Blog
3. Dev.to - Zenika
4. GitLab Blog