The integration of Docker within GitLab CI/CD pipelines represents a fundamental pillar of modern DevOps, allowing developers to package applications into immutable artifacts that ensure consistency across development, testing, and production environments. By leveraging GitLab CI/CD with Docker, organizations can automate the creation of Docker images, execute comprehensive tests within those containers, and push the finalized images to a container registry. This capability is available across all service tiers, including Free, Premium, and Ultimate, and is supported on GitLab.com, GitLab Self-Managed, and GitLab Dedicated offerings. To achieve this, the infrastructure must be specifically configured to allow the execution of Docker commands within CI/CD jobs, a requirement that typically involves the use of privileged mode or specialized executors to bridge the gap between the job container and the Docker daemon.
Infrastructure Requirements for Docker Execution
To execute Docker commands within a GitLab CI/CD pipeline, the GitLab Runner must be explicitly configured to support these operations. There are several architectural paths to achieve this, depending on the level of access and the security requirements of the environment.
The Shell Executor Configuration
One method to enable Docker commands is through the use of the shell executor. In this configuration, the GitLab Runner does not start a new container for each job but instead executes the scripts directly on the host machine's shell.
The implementation process for the shell executor involves the following steps:
- Install the GitLab Runner on the target server.
- Register the runner using the shell executor. An example registration command is:
sudo gitlab-runner register -n \ --url "https://gitlab.com/" \ --registration-token REGISTRATION_TOKEN \ --executor shell \ --description "My Runner" - Install the Docker Engine on the same server where the GitLab Runner is installed.
In this scenario, the gitlab-runner user executes the Docker commands. This requires the user to have the necessary permissions to interact with the Docker socket, typically achieved by adding the user to the docker group. The primary impact of using the shell executor is that it removes the isolation between jobs, as all jobs run on the same host, which can lead to state leakage but simplifies the ability to run Docker commands without nested virtualization.
Docker-in-Docker (DinD) Architecture
The most common approach for isolated builds is Docker-in-Docker (DinD). This is necessary because GitLab CI jobs typically run inside Docker containers themselves. Since the Docker CLI is merely a client that communicates with a daemon (dockerd), a standard container does not have access to a daemon unless one is provided.
The docker:dind image provides a running Docker daemon. In GitLab CI, this is implemented as a service. By defining the service in the .gitlab-ci.yml file, the job container can communicate with the daemon container.
An example configuration for a DinD build is as follows:
yaml
dind-build:
services:
- name: docker:dind
alias: dockerdaemon
variables:
DOCKER_HOST: tcp://dockerdaemon:2375/
The DOCKER_HOST variable tells the Docker CLI how to locate the daemon. By assigning the alias dockerdaemon, the CLI knows to route requests to that specific service. The impact of this setup is a fully isolated environment where each job has its own ephemeral Docker daemon, preventing interference between concurrent builds.
Advanced DinD Configuration and Optimization
For high-performance environments, specific configurations are required to optimize image layering and security.
The use of the overlay2 storage driver is highly recommended for improved performance. This is configured via the DOCKER_DRIVER variable. Additionally, for secure communication, TLS certificates must be managed. The DOCKER_TLS_CERTDIR variable specifies where Docker should create certificates, which are then shared between the service and the job container through volume mounts defined in the runner's config.toml.
A sophisticated .gitlab-ci.yml implementation for DinD would look like this:
yaml
image: docker:19.03.1
services:
- docker:19.03.1-dind
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
REGISTRY_GROUP_PROJECT: $CI_REGISTRY/root/gitlab-ci-dind-example
In this configuration, pinning the Docker version (e.g., 19.03.1) for both the image and the service ensures compatibility and prevents breaking changes from unexpected version updates.
Podman as a Daemonless Alternative
GitLab has begun recommending the use of Kaniko or Podman for building images to avoid the security risks associated with privileged mode. Podman is particularly attractive because it is daemonless; the CLI performs the work itself without needing a background server process.
Podman supports nearly all the same command-line options as Docker, making the transition seamless. A Podman-based build job can be defined as follows:
```yaml
stages:
- build
podman-build:
stage: build
image:
name: quay.io/podman/stable
script:
- podman login -u "$CIREGISTRYUSER" -p "$CIREGISTRYPASSWORD" "$CIREGISTRY"
- podman build -t "$CIREGISTRYIMAGE:podman" .
- podman push "$CIREGISTRY_IMAGE:podman"
```
For users who prefer using the docker command syntax, a symbolic link can be created to map docker to podman:
yaml
podman-build:
stage: build
image:
name: quay.io/podman/stable
script:
- ln -s /usr/bin/podman /usr/bin/docker
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- docker build -t "$CI_REGISTRY_IMAGE:podman" .
- docker push "$CI_REGISTRY_IMAGE:podman"
The impact of moving to Podman is the elimination of the need for privileged mode on the runner, significantly enhancing the security posture of the CI/CD infrastructure by reducing the attack surface.
The Docker Executor and Service Management
To use the Docker executor, the runner must be registered specifically for this purpose. This allows the runner to spin up containers for each job and provide additional services.
A runner can be registered with a template configuration to supply specific services. For example, if a project requires a database for testing, a template can be created:
toml
[[runners]]
[runners.docker]
[[runners.docker.services]]
name = "postgres:latest"
[[runners.docker.services]]
name = "mysql:latest"
The runner is then registered using this template:
bash
sudo gitlab-runner register \
--url "https://gitlab.example.com/" \
--token "$RUNNER_TOKEN" \
--description "docker-ruby:2.6" \
--executor "docker" \
--template-config /tmp/test-config.template.toml \
--docker-image ruby:3.3
This setup allows the job container (in this case, ruby:3.3) to access the postgres:latest and mysql:latest services during the build process, facilitating integrated testing of the application against real database instances.
Image Requirements and Registry Integration
The image keyword in .gitlab-ci.yml defines the Docker image the executor uses to run the job. By default, these images are pulled from Docker Hub, although the gitlab-runner/config.toml can be modified to use different registries or local images.
Every image used in a CI/CD job must meet minimum functional requirements to be compatible with the GitLab Runner. Specifically, the image must have the following applications installed:
shbashgrep
Without these, the runner cannot execute the script sections of the pipeline.
Container Registry Authentication
When using the GitLab Container Registry on the same instance, GitLab simplifies authentication using the CI_JOB_TOKEN. This token is automatically provided to the job. However, specific permissions are required:
- The user starting the job must have a role of Developer, Maintainer, or Owner for the project hosting the private image.
- The project hosting the private image must be configured to allow the requesting project to authenticate via the job token (this is disabled by default).
The runner determines authentication credentials by checking the following locations in order:
- A
config.jsonfile in the/root/.dockerdirectory. - A
DOCKER_AUTH_CONFIGCI/CD variable. - A
DOCKER_AUTH_CONFIGenvironment variable in the runner'sconfig.toml. - A
config.jsonfile in the$HOME/.dockerdirectory of the user running the process.
If the --user flag is used to run child processes as an unprivileged user, the home directory of the main runner process user is utilized. Note that Credential Helpers and Credentials Stores require specific binaries to be present in the GitLab Runner's $PATH.
Practical Implementation Example
To illustrate the entire process, consider a Python application. A Dockerfile for such an application might look as follows:
dockerfile
FROM python:3.9-slim-bullseye
RUN pip install cowsay
ENTRYPOINT python -c "import cowsay; cowsay.tux('hello')"
To build and push this image using a DinD configuration in GitLab CI, the following sequence is used:
- Authenticate with the registry:
docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - Build the image:
docker build -t "$CI_REGISTRY_IMAGE:dind" . - Push the image to the registry:
docker push "$CI_REGISTRY_IMAGE:dind"
Once the image is pushed to the registry (e.g., registry.gitlab.com/pythonspeed/building-docker-images:dind), it can be run manually to verify the output:
bash
docker run registry.gitlab.com/pythonspeed/building-docker-images:dind
Comparative Analysis of Execution Methods
The following table provides a detailed comparison of the various methods for running Docker within GitLab CI.
| Method | Executor | Privileged Mode Required | Daemon Requirement | Primary Advantage | Primary Disadvantage |
|---|---|---|---|---|---|
| Shell Executor | Shell | No (Host level) | Host Docker Engine | Simple setup, fast | No isolation, security risk |
| DinD | Docker | Yes | docker:dind service |
High isolation | Slow, requires privileged mode |
| Podman | Docker | No | Daemonless | Secure, no privileged mode | Slightly different CLI syntax |
| Kaniko | Docker | No | Daemonless | Recommended by GitLab | Different build syntax |
Detailed Analysis of Pipeline Security and Performance
The choice of Docker integration method has significant implications for both the security of the host system and the speed of the development cycle. Using the shell executor provides the fastest performance because it avoids the overhead of starting new containers and utilizes the host's existing Docker cache. However, it introduces a critical security vulnerability: any job can potentially access the host system's files and other jobs' data.
Conversely, Docker-in-Docker (DinD) provides an isolated sandbox for every job. While this is more secure from a data-leakage perspective, the requirement for privileged mode means the container has nearly the same access to the host kernel as a root user, which is a significant security concern in shared environments. This is why the industry is shifting toward daemonless tools like Podman and Kaniko. These tools allow for image construction without requiring the container to have root-level access to the host kernel, thereby fulfilling the principle of least privilege.
From a performance standpoint, the overlay2 driver in DinD is essential. Without it, Docker uses slower storage drivers that can increase build times by several minutes for complex images. Furthermore, the use of pinned versions for images (e.g., docker:19.03.1) is not merely a suggestion but a requirement for stable production pipelines. Relying on the latest tag can lead to "flaky" builds where a pipeline works one day and fails the next because the underlying Docker version was updated by the maintainers.