GitLab CI Docker Swarm Orchestration and Deployment

The integration of GitLab CI/CD with Docker creates a powerhouse for automated software delivery, allowing developers to move from code commit to production deployment with minimal manual intervention. GitLab, acting as an open source Git code management system comparable to platforms like Bitbucket or GitHub, provides a natively integrated Continuous Integration and Continuous Deployment (CI/CD) framework. This ecosystem allows for the creation of automated pipelines where code is tested, built into immutable artifacts, and deployed across various environments. A critical component of this architecture is the GitLab Runner, the agent that executes the jobs defined in the pipeline. When these runners are deployed within a Docker Swarm mode cluster, they enable a seamless transition from the build stage to the deployment of production stacks, leveraging the orchestrator's ability to manage containerized services across a cluster of nodes.

Architectural Strategies for GitLab Runner Deployment

The deployment of a GitLab Runner requires careful consideration of the execution environment to ensure persistence and accessibility to the Docker daemon. While it is possible to run runners on dedicated, isolated machines to maintain strict security boundaries during the build process, there is a specific strategic advantage to deploying runners within the target Docker Swarm cluster.

Integrating a runner directly into the production Docker Swarm mode cluster allows the CI/CD pipeline to execute deployment commands, such as stack updates, directly against the Swarm Manager. However, the method of deployment for the runner itself is paramount.

The Standalone Docker Approach for Runners

It is highly recommended to run the GitLab Runner in Docker standalone mode, even when that runner is hosted on a Docker Swarm mode Manager Node. This design choice is driven by the need for configuration persistence. If a GitLab Runner is deployed as a Docker Swarm mode service, the orchestrator may move the runner to a different Manager Node during a reschedule or destroy and replace the container during an update. Such an event would result in the loss of the registration configuration if not handled correctly. By using standalone Docker, the configuration persists within the created container and associated volumes, ensuring the runner remains registered and operational.

To instantiate a GitLab Runner in standalone mode, the following command is utilized:

docker run -d --name gitlab-runner --restart always -v gitlab-runner:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest

Once the container is active, the administrator must enter the container via a Bash session to perform the registration:

docker exec -it gitlab-runner bash

Runner Registration and Configuration

Registration is the process of linking the runner agent to the GitLab instance. This requires the runner to be authenticated via a specific URL and a registration token, which are obtained from the GitLab Admin Area under the Runners section.

Environment Variable Setup

Before executing the registration command, the necessary credentials should be exported as environment variables within the container session to ensure a clean and repeatable setup:

export GITLAB_URL=https://gitlab.example.com/
export GITLAB_TOKEN=WYasdfJp4sdfasdf1234

Execution of the Registration Command

The registration process defines how the runner behaves, which images it uses, and how it interacts with the host system. The following command demonstrates a professional registration setup:

gitlab-runner register -n --name "Docker Runner" --executor docker --docker-image docker:latest --docker-volumes /var/run/docker.sock:/var/run/docker.sock --url $GITLAB_URL --registration-token $GITLAB_TOKEN --tag-list dog-cat-cluster,stag,prod

The parameters used in this command have significant implications for the pipeline:

• The --executor docker flag ensures that each CI/CD job is executed within a fresh Docker container, providing isolation between builds.
• The --docker-volumes /var/run/docker.sock:/var/run/docker.sock mapping is critical; it allows the runner to communicate with the Docker daemon of the host machine, enabling the "Docker-out-of-Docker" (DooD) pattern.
• The --tag-list allows the runner to be targeted for specific jobs (e.g., staging or production) by matching the tags defined in the .gitlab-ci.yml file.

After successful registration, the runner becomes visible in the GitLab web user interface, where administrators can further modify its name, tags, and other metadata.

Docker Image Construction and Build Strategies

GitLab CI/CD supports the creation of Docker images across all tiers (Free, Premium, and Ultimate) and across all offerings, including GitLab.com, GitLab Self-Managed, and GitLab Dedicated. The primary goal is to transform source code into a Docker image, test that image, and push it to a container registry.

Privileged Mode and the Shell Executor

To execute Docker commands (like docker build or docker push) within a CI/CD job, the runner must be configured to support these commands. There are two primary paths for achieving this:

1. The Privileged Mode Path: This requires the Docker executor to run in privileged mode, granting the container access to the host's kernel features.
2. The Shell Executor Path: This bypasses the need for a Docker-in-Docker container by running commands directly on the host machine's shell.

To implement the Shell Executor, the administrator must install the GitLab Runner on a server and select the shell executor during registration:

sudo gitlab-runner register -n --url "https://gitlab.com/" --registration-token REGISTRATION_TOKEN --executor shell --description "My Runner"

For this configuration to function, the Docker Engine must be installed on the server where the runner resides, and the gitlab-runner user must be granted the necessary permissions to execute Docker commands.

Advanced Registry Authentication and Credential Management

Managing access to private registries, such as the Amazon Elastic Container Registry (ECR), requires specific configuration to avoid authentication failures. A common issue occurs when a pipeline uses both private registry images and public images from Docker Hub; if the Docker daemon attempts to use the same credentials for both, the pull from Docker Hub may fail.

Implementing Credential Helpers

To solve this, GitLab Runner can be configured to use Credential Helpers. For AWS ECR, the docker-credential-ecr-login helper must be available in the GitLab Runner's $PATH.

There are two methods to provide the necessary JSON configuration to the runner:

• Via CI/CD Variables: Create a variable named DOCKER_AUTH_CONFIG.
• Via Self-Managed Configuration: Add the JSON configuration directly to ${GITLAB_RUNNER_HOME}/.docker/config.json.

The configuration for a specific registry is defined as follows:

{ "credHelpers": { "<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login" } }

Alternatively, to apply the helper to all ECR registries, the following configuration is used:

{ "credsStore": "ecr-login" }

When using credsStore, the region must be explicitly defined in the AWS shared configuration file located at ~/.aws/config, as the helper requires the region to retrieve the authorization token.

Pipeline Implementation via .gitlab-ci.yml

The logic of the CI/CD process is governed by the .gitlab-ci.yml file, which resides in the root of the code repository. This file defines the stages, the images used for execution, and the specific scripts to be run.

Sample Pipeline Architecture

For an environment utilizing a Docker Swarm cluster with a Traefik proxy, the pipeline configuration is structured to handle both the build and the deployment phases.

```yaml
image: tiangolo/docker-with-compose

beforescript:
- docker login -u gitlab-ci-token -p $CIJOBTOKEN $CIREGISTRY

stages:
- build
- deploy

build-prod:
stage: build
script:
- docker-compose build
only:
- master

deploy-prod:
stage: deploy
script:
- docker stack deploy -c docker-compose.yml --with-registry-auth my-stack
only:
- master
```

Analysis of Pipeline Components

• Image Selection: The use of tiangolo/docker-with-compose provides an environment pre-equipped with the tools necessary to interact with Docker and Docker Compose.
• Authentication: The before_script block ensures that the runner is authenticated to the GitLab Container Registry using the CI_JOB_TOKEN, which is a short-lived token provided by GitLab for each job.
• Stage Separation: The build stage focuses on creating the image, while the deploy stage interacts with the Swarm cluster.
• Deployment Command: The command docker stack deploy -c docker-compose.yml --with-registry-auth my-stack tells the Swarm Manager to update the service based on the compose file and ensures that the worker nodes are authorized to pull the image from the registry.

Technical Specification Summary

The following table outlines the critical configurations and requirements for GitLab CI Docker deployments.

Conclusion

The deployment of applications via GitLab CI into a Docker Swarm environment represents a sophisticated orchestration of tools designed to minimize the "time to production." By strategically deploying the GitLab Runner in standalone mode on a Swarm Manager node, organizations can ensure that their deployment pipelines have the necessary permissions and persistence to manage production stacks without the risks associated with ephemeral Swarm services. The transition from the build stage, where images are constructed and pushed to a registry, to the deployment stage, where docker stack deploy is invoked, creates a reliable and repeatable path to software delivery. Furthermore, the implementation of credential helpers for services like AWS ECR ensures that the pipeline can securely access private images without compromising security or stability. This holistic approach—combining the flexibility of the .gitlab-ci.yml definition with the power of Docker Swarm's orchestration—allows for a scalable, industrial-grade CI/CD pipeline that is capable of handling complex microservices architectures.

Sources

1. DockerSwarm.rocks
2. GitLab Documentation - Using Docker to build Docker images
3. GitLab Documentation - Running CI/CD jobs in Docker containers