GitLab CI/CD Containerization and the Docker-in-Docker Architecture

The integration of Docker within GitLab CI/CD pipelines represents a cornerstone of modern DevOps, providing a standardized mechanism for encapsulating application environments and automating the software delivery lifecycle. At its core, the ability to execute CI/CD jobs within Docker containers ensures that every stage of the pipeline—from linting and unit testing to integration and deployment—occurs in a consistent, reproducible environment. This eliminates the "it works on my machine" phenomenon by ensuring that the build environment is identical for every developer and every pipeline execution. This capability is available across all GitLab tiers, including Free, Premium, and Ultimate, and is supported across GitLab.com, GitLab Self-Managed, and GitLab Dedicated offerings.

To achieve a fully containerized pipeline, GitLab employs the GitLab Runner, an agent that manages the execution of jobs. When configured with the Docker executor, the runner spawns a fresh container for every job, based on a specified image. This isolation prevents side effects from one job from affecting another and allows different jobs in the same pipeline to use entirely different operating systems or language runtimes. However, a complex challenge arises when the goal of a CI/CD job is not just to run inside a container, but to build, test, and push a new Docker image to a registry. This requirement necessitates a "Docker-in-Docker" (DinD) or a similar orchestration strategy, as the standard container environment does not inherently possess the Docker daemon required to execute docker build or docker push commands.

Implementing CI/CD Jobs in Docker Containers

To initiate the process of running CI/CD jobs within Docker containers, a specific architectural setup is required. The primary requirement is the registration of a GitLab Runner configured specifically to use the Docker executor. This executor tells the runner to pull a specified image from a registry and start a container to host the job's shell.

In the .gitlab-ci.yml configuration file, the user must specify the container image where the jobs will execute. This is a critical step as it defines the toolset available to the job. For instance, a job requiring Python 3.11 and Pytest would specify a corresponding Python image. Additionally, GitLab allows for the optional execution of "services." Services are additional containers that run alongside the main job container, facilitating the testing of applications that depend on external databases or caches, such as MySQL or Redis.

The operational flow for this setup involves:

• Registering a runner and configuring it to use the Docker executor.
• Defining the image in the .gitlab-ci.yml file.
• Optionally defining services to provide external dependencies.

The impact of this architecture is a drastic reduction in environment drift. By specifying the image in the configuration file, the infrastructure becomes version-controlled, meaning any change to the build environment is tracked via Git commits.

The Docker-in-Docker (DinD) Paradigm

Docker-in-Docker, frequently abbreviated as dind, is a specific configuration where the Docker executor uses a container image of Docker to run the job script. This allows the job to have its own isolated Docker daemon running inside the container.

For a successful DinD implementation, the following conditions must be met:

• The registered runner must use either the Docker executor or the Kubernetes executor.
• The executor must use a Docker-provided image that contains all necessary Docker tools.
• The job must be run in privileged mode.

One of the most critical security considerations with DinD is the requirement for privileged mode. Enabling --docker-privileged effectively disables the container's security mechanisms. This creates a significant risk of container breakout, where a process inside the container could potentially gain access to the host machine's kernel or filesystem, leading to privilege escalation.

To mitigate some of the risks and ensure stability, the following best practices are mandatory:

• Use TLS enabled for the Docker daemon, which is the default for Docker 19.03.12 and later and is supported by GitLab.com instance runners.
• Pin a specific version of the Docker image. For example, using docker:24.0.5 instead of docker:latest is essential. The use of the latest tag is strongly discouraged because it removes control over the version being used, which can lead to unpredictable incompatibility problems when new Docker versions are released.

Alternative Strategies for Building Docker Images

Due to the security risks associated with privileged mode and the Docker daemon's root privileges, alternatives to the standard DinD approach are often sought.

The Shell Executor Approach

One method to enable Docker commands in CI/CD jobs without the complexities of DinD is to use the shell executor. In this configuration, the runner does not start a new container for each job but instead executes scripts directly on the host machine's shell.

To implement this, the following steps are required:

• Install the GitLab Runner on a server.
• Register the runner and select the shell executor during the process. An example registration command is:
  sudo gitlab-runner register -n --url "https://gitlab.com/" --registration-token REGISTRATION_TOKEN --executor shell --description "My Runner"
• Install the Docker Engine on the same server where the GitLab Runner is installed.

Under this setup, the gitlab-runner user executes the Docker commands. However, for this to work, the gitlab-runner user must have permission to access the Docker socket. Adding the gitlab-runner user to the docker group is a common solution, but this carries a heavy security implication: granting a user access to the Docker group is effectively granting them full root permissions on the host.

Podman as a Secure Alternative

Podman serves as a powerful alternative to Docker, specifically designed to address the security concerns of privileged mode. Unlike Docker, Podman does not rely on a central daemon and can run in rootless mode. This means that the need for privileged containers in GitLab CI/CD is eliminated, removing the primary attack vector that makes Docker daemons a target for attackers. This is particularly useful in environments deploying services to Kubernetes, such as AWS EKS, where security constraints on the runner are stringent.

Registry Authentication and Credential Management

Managing access to private container registries is a complex aspect of the GitLab CI/CD ecosystem. When using the GitLab Container Registry on the same instance, GitLab simplifies this by providing default credentials via the CI_JOB_TOKEN. However, this token requires that the user starting the job possesses a Developer, Maintainer, or Owner role, and the project hosting the private image must explicitly allow authentication via the job token, as this is disabled by default.

For external registries, such as Amazon ECR, GitLab supports Credential Helpers and Credential Stores.

Configuration for Credential Helpers

To use a private image from a registry like <aws_account_id>.dkr.ecr.<region>.amazonaws.com/private/image:latest, the following configuration is necessary:

• Ensure that the docker-credential-ecr-login binary is available in the GitLab Runner's $PATH.
• Configure the necessary AWS credentials. The GitLab Runner Manager acquires these credentials and passes them to the runners.

To make the GitLab Runner utilize these helpers, the system reads configuration in a specific order of precedence:

• A config.json file located in the /root/.docker directory.
• A DOCKER_AUTH_CONFIG CI/CD variable.
• A DOCKER_AUTH_CONFIG environment variable defined in the runner's config.toml file.
• A config.json file in the $HOME/.docker directory of the user running the process.

If the --user flag is used to run child processes as an unprivileged user, the home directory of the main runner process user is utilized.

Handling Mixed Registries

A known limitation occurs when users attempt to pull images from both a private registry and a public registry (like Docker Hub). If the credsStore is used to access all registries, pulling from Docker Hub may fail because the Docker daemon attempts to apply the same credentials to all registries regardless of the source. To resolve this, the DOCKER_AUTH_CONFIG variable can be used with specific JSON content, such as:

{ "credsStore": "osxkeychain" }

This ensures that the specific credential helper is invoked correctly based on the registry being accessed.

Lightweight Dockerized GitLab Setup on Ubuntu

For users who require a streamlined development workflow without the overhead of a massive enterprise installation, a lightweight, Dockerized GitLab instance can be deployed on an Ubuntu server. This approach leverages the encapsulation properties of Docker to simplify the deployment of the GitLab application itself.

This specific architectural pattern typically involves the following components:

• Docker Compose: Used to orchestrate the GitLab container and its associated services.
• Nginx: Acts as a reverse proxy to handle incoming traffic and route it to the GitLab container.
• Certbot: Used to automate the acquisition and renewal of SSL/TLS certificates for secure HTTPS communication.
• GitLab Runner: Installed as a separate agent to handle the automated builds, tests, and deployments.

By using Docker Compose, the entire GitLab environment—including the database and the application server—is treated as a single unit, making it easier to migrate, back up, and update.

Comparative Summary of Execution Methods

The following table provides a technical comparison of the different methods for running Docker commands within GitLab CI/CD.

Conclusion

The deployment of Docker within GitLab CI/CD is a nuanced balance between operational flexibility and security. While Docker-in-Docker provides the most direct path to building images, its reliance on privileged mode introduces significant vulnerabilities that can expose the host system to compromise. The move toward rootless alternatives like Podman and the use of specific credential helpers for registries like AWS ECR demonstrate an evolution toward more secure, "least-privilege" architectures.

The choice of executor—whether Shell, Docker, or Kubernetes—dictates the level of isolation and the method of authentication required for container registries. For most users, the Docker executor combined with a pinned version of the docker:dind image and TLS encryption provides the best compromise of usability and reliability. However, for organizations with strict security compliance requirements, the transition to Podman or the use of the Shell executor with strictly controlled user permissions is necessary. Ultimately, the goal of these configurations is to create a seamless bridge between the source code and the final container image, ensuring that the transition from a commit to a deployed artifact is automated, secure, and repeatable.

Sources

1. Using Docker images in GitLab CI/CD
2. Using Docker to build Docker images
3. Removing privileged mode with Podman
4. Lightweight Dockerized GitLab Setup