Docker Integration within GitLab CI/CD Pipelines

The orchestration of containerized environments within a Continuous Integration and Continuous Deployment (CI/CD) pipeline is a cornerstone of modern software engineering. Utilizing Docker within GitLab CI allows developers to encapsulate their application environments, ensuring that the build, test, and deployment phases are executed in a consistent, reproducible manner. By defining the execution environment in a .gitlab-ci.yml file, teams can eliminate the "it works on my machine" phenomenon, shifting the focus from infrastructure debugging to feature development. This integration involves a complex interplay between the GitLab Runner, the Docker executor, and the specific Docker-in-Docker (DinD) service, all of which must be meticulously configured to allow the creation and management of container images during the pipeline lifecycle.

Foundational Concepts of Docker Images in GitLab CI

The image keyword in a .gitlab-ci.yml file defines the specific Docker image that the GitLab Runner's Docker executor will pull and use to execute the jobs. This serves as the runtime environment for every script command defined in the pipeline.

By default, the Docker executor attempts to pull images from Docker Hub. However, enterprise environments often require custom registry locations, which can be configured via the gitlab-runner/config.toml file. This allows for the use of local images or private registries, reducing reliance on external networks and improving build speeds.

The image name must follow specific formats to be recognized by the executor:

• image: <image-name>: This format is equivalent to using the <image-name> with the latest tag.
• image: <image-name>:<tag>: This allows for pinning a specific version, such as docker:19.03.1, ensuring build stability across different pipeline runs.
• image: <image-name>@<digest>: This specifies a unique immutable identifier for the image, providing the highest level of security and reproducibility.

For an image to be compatible with a CI/CD job, it must meet minimum functional requirements. Specifically, the image must have the following applications installed:

• sh or bash
• grep

Without these shell utilities, the GitLab Runner cannot execute the script blocks defined in the YAML configuration, leading to immediate job failure.

Implementing Docker-in-Docker (DinD) for Image Construction

When a pipeline needs to run Docker commands—such as docker build, docker push, or docker compose—it requires a Docker daemon to be available. Since the job itself is running inside a container, the most common approach is to use Docker-in-Docker (DinD).

To implement DinD, the .gitlab-ci.yml must specify both a Docker image and a corresponding Docker service. For example, a configuration might use image: docker:19.03.1 and services: - docker:19.03.1-dind. The service launches a secondary container that runs the Docker daemon, which the primary job container communicates with to execute Docker commands.

The configuration of DinD requires specific variables to optimize performance and security:

• DOCKER_DRIVER: overlay2: Using the overlay2 storage driver is highly recommended for improved performance and efficient layer management.
• DOCKER_TLS_CERTDIR: "/certs": This variable specifies where Docker will automatically create certificates on boot. These certificates are shared between the service and the job container via a volume mount defined in the config.toml of the runner.

In certain architectures, such as Kubernetes runners, the DOCKER_HOST variable may need to be explicitly set to tcp://docker:2375. However, this is typically only necessary for Kubernetes-based deployments and not for standard Docker executor setups.

GitLab Runner Configuration for Docker Support

The ability to execute Docker commands is not solely dependent on the .gitlab-ci.yml file; it requires the underlying GitLab Runner to be configured with the correct permissions.

The Privileged Mode Requirement

To enable Docker commands in CI/CD jobs, the GitLab Runner must be configured to support them. This typically requires the runner to operate in "privileged mode." Privileged mode grants the container access to the host's kernel features, which is necessary for the Docker daemon to start inside a container.

If privileged mode is not enabled on the runner, the docker build or docker run commands will fail with permission errors. For users who cannot enable privileged mode for security reasons, Docker alternatives (such as Kaniko or Buildah) should be utilized.

Registering a Runner with the Docker Executor

To set up a runner that supports Docker and specific services, the registration process must define the executor as docker. An example of a temporary template configuration for services is as follows:

toml [[runners]] [runners.docker] [[runners.docker.services]] name = "postgres:latest" [[runners.docker.services]] name = "mysql:latest"

This template can be used during registration via the following command:

bash sudo gitlab-runner register \ --url "https://gitlab.example.com/" \ --token "$RUNNER_TOKEN" \ --description "docker-ruby:2.6" \ --executor "docker" \ --template-config /tmp/test-config.template.toml \ --docker-image ruby:3.3

In this scenario, the registered runner uses ruby:3.3 as the base image, while also spinning up postgres:latest and mysql:latest as accessible services during the build process.

Using the Shell Executor Alternative

An alternative to the Docker executor is the shell executor. In this setup, Docker commands are executed directly on the host machine where the GitLab Runner is installed.

To implement this:
1. Install the GitLab Runner on the server.
2. Install the Docker Engine on the same server.
3. Register the runner using the shell executor:

bash sudo gitlab-runner register -n \ --url "https://gitlab.com/" \ --registration-token REGISTRATION_TOKEN \ --executor shell \ --description "My Runner"

In the shell executor model, the gitlab-runner user must be granted permission to execute Docker commands (e.g., by adding the user to the docker group).

Advanced Configuration and Optimization

For high-performance environments, optimizing how images are pulled and managed is critical to reducing pipeline latency.

Registry Mirrors and Performance

To avoid Docker Hub rate limits and improve download speeds, registry mirrors can be configured. This can be achieved by appending CLI flags to the dind service in the .gitlab-ci.yml file:

yaml services: - name: docker:24.0.5-dind command: ["--registry-mirror", "https://registry-mirror.example.com"]

Alternatively, this can be configured globally in the config.toml for both Docker and Kubernetes executors:

toml [[runners]] ... executor = "docker" [runners.docker] privileged = true [[runners.docker.services]] name = "docker:24.0.5-dind" command = ["--registry-mirror", "https://registry-mirror.example.com"]

For an even more integrated approach, a /opt/docker/daemon.json file can be created on the host with the following content:

json { "registry-mirrors": [ "https://registry-mirror.example.com" ] }

The config.toml file should then be updated to mount this file to /etc/docker/daemon.json, ensuring the mirror is applied to every container created by the runner.

Practical Implementation Examples

Building Custom Runner Images

In complex DevOps workflows, it is often necessary to build a custom Docker image that includes specific tools (like the AWS CLI or ECR helpers) to be used as a runner. This requires a multi-stage build process within the pipeline.

A typical configuration for building a custom runner image includes variables for versioning and the use of the overlay2 driver:

```yaml
variables:
DOCKERDRIVER: overlay2
IMAGENAME: $CIREGISTRYIMAGE:$CICOMMITREFNAME
GITLABRUNNERVERSION: v17.3.0
AWSCLI_VERSION: 2.17.36

stages:
- build

build-image:
stage: build
script:
- echo "Logging into GitLab container registry..."
- docker login -u $CIREGISTRYUSER -p $CIREGISTRYPASSWORD $CIREGISTRY
- echo "Building Docker image..."
- docker build --build-arg GITLABRUNNERVERSION=${GITLABRUNNERVERSION} --build-arg AWSCLIVERSION=${AWSCLI 대해서VERSION} -t ${IMAGENAME}
```

The image construction process often involves installing specific dependencies such as jq, procps, curl, unzip, groff, libgcrypt20, tar, gzip, less, and openssh-client, followed by cleaning the apt cache to reduce image size:

bash apt-get update && apt-get install -y jq procps curl unzip groff libgcrypt20 tar gzip less openssh-client && apt-get clean && rm -rf /var/lib/apt/lists/*

Implementing Docker Compose in CI/CD

For developers looking to build and deploy multi-container applications, docker compose can be utilized within the pipeline. This requires the docker:latest image and the docker:dind service.

An example pipeline for building and deploying via compose is structured as follows:

```yaml
image: docker:latest
services:
- docker:dind

stages:
- build
- deploy

build:
stage: build
script:
- docker compose build

deploy:
stage: deploy
script:
- docker compose up -d
```

In this workflow, the build stage ensures the containers are constructed from the Dockerfile, and the deploy stage brings the services up in detached mode.

Detailed Configuration Mapping

The following table provides a technical breakdown of the critical components used in GitLab CI Docker configurations.

Analysis of Deployment Strategies

The transition from building an image to deploying it involves several critical steps that vary based on the target environment. While building an image is a standardized process using docker build and docker push to the GitLab Container Registry, the deployment phase is highly dependent on the destination.

For cloud deployments, the pipeline must integrate with provider-specific APIs (such as AWS, GCP, or Azure). This often involves using custom images that include the AWS CLI and ECR Credential Helper to authenticate and push images to private registries.

The use of the $CI_REGISTRY variable allows the pipeline to dynamically target the GitLab internal registry, while variables like $CI_REGISTRY_USER and $CI_REGISTRY_PASSWORD provide the necessary credentials for authentication during the docker login phase. This ensures that the pipeline can securely push built images without hardcoding sensitive information.

Conclusion

The integration of Docker within GitLab CI/CD is a multifaceted process that requires alignment between the .gitlab-ci.yml configuration, the GitLab Runner's config.toml, and the host's environment. By leveraging the Docker executor and the Docker-in-Docker (DinD) service, organizations can achieve a high degree of automation in their build and deployment cycles. The reliance on overlay2 for performance and privileged mode for functionality underscores the technical requirements of containerized orchestration. Furthermore, the flexibility to switch between the Docker executor and the shell executor, or to optimize image pulling via registry mirrors, allows for a scalable CI/CD architecture that can adapt to various security and performance constraints. Ultimate success in this configuration depends on the precise pinning of image versions and the correct mapping of volumes for TLS certificates, ensuring a secure and stable pipeline.

Sources

1. gitlab-ci-dind-example GitHub Repository
2. GitLab Docs - Using Docker Images
3. GitLab Docs - Using Docker Build
4. GitLab Forum - How to write a gitlab-ci.yml file