Architecting Containerized Build Environments via GitLab Runner and Docker-in-Docker

The orchestration of modern software delivery pipelines necessitates a robust mechanism for creating, testing, and distributing containerized applications. Within the GitLab ecosystem, the GitLab Runner serves as the fundamental execution agent, responsible for picking up jobs defined in the .gitlab-ci.yml configuration and running them in specified environments. When a development lifecycle requires the creation of Docker images—for instance, packaging an application, running integration tests against that image, and subsequently pushing the artifact to a container registry—the Runner itself must possess the capability to execute Docker commands. This requirement introduces a complex architectural decision point: how to provide a containerized environment with the authority to manipulate the Docker engine. This challenge is typically addressed through three primary methodologies: the Shell executor, Socket Passthrough, and Docker-in-Docker (DinD). Each approach carries distinct implications for security, isolation, performance, and operational complexity, particularly when navigating the nuances of privileged mode and TLS encryption.

Architectural Methodologies for Docker Execution in GitLab CI/CD

To facilitate the execution of docker build, docker run, and docker push within a CI/CD pipeline, the GitLab Runner must be configured to support these commands. The choice of architecture dictates the level of isolation provided to the job and the security posture of the host machine.

The Shell Executor Approach

The Shell executor is the most direct method for enabling Docker commands. In this configuration, the GitLab Runner is installed directly on a host operating system, and the jobs are executed as processes on that host.

When a job is triggered, the gitlab-runner user executes the commands directly in the host's shell. For this to work, the user running the GitLab Runner must be granted explicit permission to interact with the Docker Engine on the host.

The primary impact of this method is a significant reduction in isolation. Because the commands run directly on the host, a malicious or poorly written script in a CI/CD job has direct access to the host's shell environment. This bypasses the containerization layer that typically protects the host from the job.

To implement this, one must first install the GitLab Runner on the target server and then install the Docker Engine on that same server. The registration process follows a specific command structure:

bash sudo gitlab-runner register -n \ --url "https://gitlab.com/" \ --registration-token REGISTRATION_TOKEN \ --executor shell \ --description "My Runner"

Socket Passthrough (Docker Socket Binding)

Socket Passthrough is a hybrid approach designed to provide the benefits of containerization while allowing the job container to interact with the host's Docker daemon. In a standard Docker-based Runner, the Runner container contains the Docker CLI tools but lacks the Docker daemon itself. Without a daemon, the CLI tools have no engine to communicate with, resulting in errors such as Cannot connect to the Docker daemon at tcp://docker:2375.

Socket Passthrough resolves this by mounting the host's Docker socket (/var/run/docker.sock) into the GitLab Runner container. This allows the Docker CLI inside the job container to send instructions directly to the Docker daemon running on the host machine.

The real-world consequence of this method is that the "containerized" job is actually controlling the host's engine. This enables features like bind-mounts to work seamlessly, as the Docker daemon is the host's daemon. A common configuration for a high-performance, socket-passthrough Runner involves modifying the config.toml file to include the socket in the volumes list.

A typical [[runners]] configuration for socket passthrough is as follows:

```toml
[[runners]]
name = "docker-socket"
url = "https://"
token = ""
executor = "docker"

[runners.docker]
tlsverify = false
image = "docker:edge-git"
privileged = false
disableentrypointoverwrite = false
oomkilldisable = false
disablecache = false
volumes = ["/builds:/builds", "/cache", "/var/run/docker.sock:/var/run/docker.sock"]
```

Docker-in-Docker (DinD) and the Privileged Mode Requirement

Docker-in-Docker (DinD) is a method where a Docker daemon runs entirely inside a container. This provides the highest level of isolation between the job and the host, as the job's Docker daemon is a completely separate instance from the host's daemon. However, this architecture comes with a severe security trade-off.

To run a Docker daemon inside a container, the container must be started in --privileged mode. Enabling privileged mode effectively disables the container's security mechanisms and exposes the host to potential privilege escalation. This can lead to "container breakout," where an attacker gains control over the underlying host machine.

Because TLS is the default in Docker versions 19.03.12 and later, configuring DinD requires careful handling of certificates to ensure the Docker client can securely communicate with the internal Docker daemon.

The registration command for a DinD runner requires the --docker-privileged flag and a volume mount for the necessary TLS certificates:

bash sudo gitlab-runner register -n \ --url "https://gitlab.com/" \ --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ --tag-list "tls-docker-runner" \ --docker-image "docker:24.0.5-cli" \ --docker-privileged \ --docker-volumes "/certs/client"

In this setup, the --docker-privileged flag ensures the container has the Linux capabilities required to manage network interfaces and file systems necessary for a daemon. The --docker-volumes "/certs/client" flag is critical; it mounts the directory where certificates are stored, allowing the Docker client within the build container to authenticate with the service container.

GitLab Runner Deployment and Container Management

When deploying GitLab Runner itself as a Docker container, specific operational considerations must be addressed to ensure persistence and scalability.

Container Lifecycle and Persistence

Running the gitlab-runner inside a Docker container requires a strategy for configuration persistence. If the container is restarted or recreated, any local changes to the configuration will be lost unless a permanent volume is mounted.

The standard procedure involves pulling the image and running it with a volume mount for the configuration. The command structure for running the container is:

bash docker run -d [options] <image-uri> <runner-command>

For example, to ensure the configuration is preserved, one should use:

bash docker run -d -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:latest

Advanced Configuration Options

The GitLab Runner Docker image is highly versatile, offering several flags to tailor the environment to specific deployment needs:

• Timezone Management: The container's timezone can be adjusted using the --env TZ=<TIMEZONE> flag. This is essential for ensuring that logs and job timestamps align with organizational standards.
• Session Server: If the Runner is configured to use a session_server, port 8093 must be exposed using the -p 8093:8093 flag in the docker run command.
• Docker Machine for Autoscaling: For environments utilizing the Docker Machine executor to scale runners dynamically, the Docker Machine storage path must be persisted. This can be achieved through system volume mounts:
  • -v /srv/gitlab-runner/docker-machine-config:/root/.docker/machine
  • Or via Docker named volumes: -v docker-machine-config:/root/.docker/machine

Image Compatibility and Base Layers

The GitLab Runner Docker images are built using either Ubuntu or Alpine Linux as their base operating systems. These images are designed to wrap the standard gitlab-runner command, providing a seamless experience that mirrors a direct host installation.

An important technical detail regarding versioning is that the GitLab Runner images maintain both backwards and forwards compatibility with the Docker Engine. This means the version of the Docker Engine installed on the host does not need to match the version used within the Runner container or the job containers. This flexibility allows administrators to update the host's Docker Engine to the latest stable version to ensure security updates without fear of breaking the CI/CD pipeline.

To interact with the Runner container directly for troubleshooting, one can use the following command pattern:

bash docker run --rm -t -i gitlab/gitlab-runner --help

This command executes the container, provides an interactive terminal, and immediately exits after displaying the help information, which is useful for verifying the installation and version (e.g., VERSION: 18.10.1).

Integration Testing and Pipeline Automation

The ultimate goal of configuring a Docker-capable Runner is to enable automated workflows. A common use case involves a CI pipeline that builds an application image, performs integration tests by mounting data volumes, and pushes the final image to a registry.

Example Pipeline Workflow

A typical .gitlab-ci.yml file for a project requiring Docker builds might look like this:

```yaml
image: docker:20.10.16
variables:
DOCKERREGISTRY: my-docker-registry.com
DOCKERREGISTRYUSER: gitlab
DOCKERIMAGE: ${DOCKER_REGISTRY}/my-app

beforescript:
- docker login -u ${DOCKERREGISTRYUSER} -p ${CIBUILDTOKEN} ${DOCKERREGISTRY}

buildandtest:
script:
- docker build -t ${DOCKERIMAGE} -f Dockerfile .
- docker run --rm -v$(pwd)/test/int-00:/app/data ${DOCKERIMAGE} npm test
- docker push ${DOCKER_IMAGE}
```

In this scenario, the docker run command utilizes a bind-mount (-v$(pwd)/test/int-00:/app/data) to inject test data into the container being tested. This allows for rapid integration testing of the application in various states without requiring external data dependencies. This level of automation is only possible if the Runner has been correctly configured via one of the three methods discussed previously.

Comparative Analysis of Execution Strategies

Selecting the correct execution strategy requires a balance of the following technical factors:

The decision-making process must prioritize the security of the host infrastructure. While the Shell executor is simple, its lack of isolation makes it unsuitable for multi-tenant environments. Socket Passthrough offers a middle ground, providing containerized jobs that share the host's engine, which is highly efficient but requires trust in the job's ability to interact with the daemon. Docker-in-Docker provides the cleanest separation of concerns but introduces the significant risk of privilege escalation due to the mandatory use of privileged mode.

Technical Implications of Runner Deployment

The deployment of GitLab Runner within a containerized environment essentially delegates full control over the Docker daemon to each container. This has a profound impact on the security guarantees of the system. If a GitLab Runner is running inside a Docker daemon that is also hosting other sensitive payloads, the isolation guarantees are effectively broken. Every command executed by the Runner is the functional equivalent of a docker run command on the host.

For instance, a command initiated within the Runner:
gitlab-runner <command>

Is equivalent to the following on the host:
docker run <options> gitlab/gitlab-runner <command>

This realization is critical for DevOps engineers. It means that the security boundary is not the container itself, but the underlying Docker daemon and the permissions granted to the Runner. If the Runner is configured with socket passthrough, the job has the power to stop, delete, or create any container on the host. If it is configured with DinD and privileged mode, the job has even deeper access to the host's kernel and hardware resources.

Sources

1. GitLab Documentation: Use Docker to build Docker images
2. Hiebl Blog: GitLab Runner Docker in Docker
3. GitLab Documentation: Install GitLab Runner Docker