Orchestrating GitLab Runner within Rancher Desktop and Kubernetes Environments

The integration of GitLab Runner into a containerized orchestration environment represents a critical junction in modern DevOps workflows. As organizations transition from monolithic CI/CD pipelines to highly distributed, scalable microservices architectures, the method by which execution jobs are processed becomes a primary determinant of engineering velocity. Deploying a GitLab Runner within a Rancher-managed Kubernetes cluster or a local Rancher Desktop environment introduces unique layers of complexity, involving container orchestration, secret management, and resource allocation. This technical exposition details the architectural nuances, deployment methodologies, and troubleshooting imperatives associated with running GitLab Runner in such ecosystems.

GitLab Runner serves as the execution engine for the jobs defined within a .gitlab-ci.yaml file. It is distinct from the GitLab Kubernetes agent, which is designed to deploy containers into Kubernetes clusters for purposes such as end-to-end (e2e) testing and continuous delivery. While the agent facilitates the deployment of applications, the Runner facilitates the execution of the logic required to build, test, and deploy those applications. For optimal security and performance, it is a fundamental best practice to install the GitLab Runner on a machine or cluster separate from the primary GitLab instance hosting the repositories.

Architectural Foundations and Prerequisites

Before initiating a deployment within a Kubernetes-based environment like Rancher, several structural components must be synchronized. The deployment process relies heavily on the Helm package manager to orchestrate the complex configuration of pods, services, and service accounts required by the Runner.

The following table outlines the essential tools and prerequisites required for a successful deployment in a cluster managed by Rancher or similar tools.

Tool / Requirement Purpose Installation Context
GitLab Repository The source of truth containing .gitlab-ci.yaml Remote or Self-Managed
Kubernetes Namespace The logical isolation layer for the runner Cluster-side (Rancher GUI)
kubectl The command-line interface for cluster interaction Local Machine
helm The package manager for Kubernetes orchestration Local Machine
kubeseal Secure management of sensitive GitLab tokens Local Machine / Cluster
kubeconfig Authentication credentials for cluster access Local Machine

The selection of a Kubernetes namespace is a critical first step. Users must identify or create a specific namespace where the Runner's pods will reside. This isolation ensures that the Runner's resource consumption and service accounts do not bleed into other project environments. Within a Rancher-managed environment, the availability of these namespaces can be verified through the Rancher Graphical User Interface (GUI).

To facilitate interaction with the cluster, the kubectl tool must be present on the local workstation, coupled with a properly configured kubeconfig file. Without this file, the local machine lacks the necessary identity and permissions to issue commands to the Rancher-managed API server. Furthermore, since the GitLab Runner is distributed as a Helm chart, helm is a non-negotiable requirement for deployment and lifecycle management.

Secure Secret Management via Sealed Secrets

In a GitOps-driven workflow, managing sensitive information such as GitLab registration tokens presents a significant security risk. Storing these tokens in plain text within a repository or a values.yaml file violates fundamental security principles. To mitigate this, the use of kubeseal is recommended.

kubeseal allows for the creation of Sealed Secrets, which are encrypted versions of Kubernetes Secrets that can be safely stored in a Git repository. Only the controller running inside the Kubernetes cluster possesses the private key required to decrypt and apply these secrets. This mechanism ensures that even if a repository is compromised, the actual registration tokens remain encrypted and useless to unauthorized actors. The deployment workflow involves:

  1. Generating a standard Kubernetes Secret containing the GitLab token.
  2. Using kubeseal to encrypt that secret into a SealedSecret custom resource.
  3. Committing the SealedSecret to the version control system.
  4. The cluster controller automatically decrypts the secret into a usable Secret object when applied.

Deployment via Helm Orchestration

The official and most robust method for deploying GitLab Runner into a Kubernetes cluster is utilizing the GitLab Runner Helm chart. This chart is specifically designed to configure the Runner to use the Kubernetes executor, which provisions a fresh pod for every individual CI/CD job. This ephemeral nature of the pods ensures a clean, isolated environment for every build, preventing state leakage between jobs.

Repository Initialization and Versioning

Before installation, the local Helm environment must be aware of the GitLab chart repository. The following commands are used to add the repository and inspect available versions.

bash helm repo add gitlab https://charts.gitlab.io

Because GitLab Runner and its Helm charts do not follow a synchronized versioning scheme, it is imperative to verify which chart version corresponds to the desired GitLab Runner application version. The user must execute a search to map these versions accurately.

For users utilizing Helm 3:
bash helm search repo -l gitlab/gitlab-runner

The output of this command provides a mapping such as the following:

NAME CHART VERSION APP VERSION DESCRIPTION
gitlab/gitlab-runner 0.64.0 16.11.0 GitLab Runner
gitlab/gitlab-runner 0.63.0 16.10.0 GitLab Runner
gitlab/gitlab-runner 0.62.1 16.9.1 GitLab Runner
gitlab/gitlab-runner 0.62.0 16.9.0 GitLab Runner

Configuration via values.yaml

The values.yaml file serves as the primary configuration interface for the Helm deployment. It allows the user to tailor the Runner to the specific resource constraints and operational requirements of the Rancher cluster.

Key considerations for the values.yaml file include:

  • Version Alignment: The Runner version should match the GitLab instance version to prevent compatibility regressions.
  • Image Availability: If a specified version of the Runner image is not available in the container registry, the deployment will result in an ImagePullBackOff error.
  • Resource Management: Users must adjust CPU and memory limits within the file to stay within the project's allocated quotas in the Kubernetes cluster.
  • Caching Mechanisms: To prevent slow job execution due to constant dependency downloading, a distributed cache (such as an S3-compatible storage like CESNET S3) or a shared volume must be configured within the values.yaml.

Execution of the Installation

Once the values.yaml is meticulously prepared, the deployment is initiated using the helm install command. This command applies the configuration to the specified namespace.

bash helm install -n <YOUR NAMESPACE> gitlab-runner -f values.yaml gitlab/gitlab-runner

In this command, <YOUR NAMESPACE> must be replaced with the actual name of the namespace where the Runner is intended to reside.

Lifecycle Management and Upgrades

Maintaining a GitLab Runner requires periodic updates to ensure security patches and new features are integrated. Upgrading a Runner in a Kubernetes environment is not a simple overwrite; it requires a controlled process to prevent job interruptions or authorization failures.

The Upgrade Workflow

To perform a safe upgrade using helm upgrade, the following prerequisites must be met:

  • The existing GitLab Runner must be paused within the GitLab UI. This ensures that no new jobs are picked up during the transition and that currently running jobs do not fail due to sudden changes in the runner's identity or configuration.
  • All currently executing jobs must be allowed to complete.

The upgrade command is executed as follows:

bash helm upgrade --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> <RELEASE-NAME> gitlab/gitlab-runner

Where <NAMESPACE> is the target namespace, <CONFIG_VALUES_FILE> is the path to the updated configuration file, and <RELEASE-NAME> is the name assigned to the runner during the initial installation.

Troubleshooting Local Environments: The Rancher Desktop and WSL2 Conflict

A significant challenge has been identified when attempting to run GitLab Runners in local development environments using Rancher Desktop on Windows via WSL2. While runners typically function without issue on macOS (using Docker Desktop) or standard Windows/WSL2 setups (using Docker Desktop), a specific failure mode occurs when Rancher Desktop is the container engine.

Analysis of Error Code 143

Users have reported that pipelines executed via a local GitLab Runner in a Rancher Desktop environment frequently fail with error code 143.

The error code 143 in a Linux/Container context typically corresponds to a SIGTERM signal. This indicates that the process was terminated by the operating system or the container engine. In the context of Rancher Desktop on WSL2, this failure often occurs during the "Preparing environment" or "Pulling docker image" stages.

A typical reproduction scenario involving a bash script for local runner registration is outlined below:

```bash

!/bin/bash

Creates a volume to store the runner config

docker volume create local-gitlab-runner-config

Creates a runner

docker run --rm --name local-gitlab-runner -d \
-v local-gitlab-runner-config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

Registers the runner and stores the config in the volume

docker run --rm -v local-gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner register \
--non-interactive \
--url "" \
--token "" \
--executor "docker" \
--docker-image alpine:latest
```

In this scenario, the runner is registered successfully and appears online in the GitLab interface. However, when a job is triggered, the Docker executor attempts to pull the specified image (e.g., registry.gitlab.com/gitlab-org/terraform-images/releases/1.4:v1.0.0) and prepare the environment. The failure with error code 143 suggests that the container runtime or the WSL2 integration is terminating the container before the job can complete. This discrepancy between Docker Desktop and Rancher Desktop highlights underlying differences in how the container runtimes (such as containerd vs. dockerd) and the WSL2 kernel interact with the host OS, particularly regarding signal handling and resource management.

Comparative Analysis of Runner Implementations

Understanding the distinction between various runner deployment methods is vital for selecting the correct strategy for a given infrastructure.

Feature Local Docker Runner Kubernetes Executor (Helm)
Scaling Mechanism Manual or Docker Swarm Automatic via Kubernetes Pods
Isolation Level Container-level Pod-level (Namespace isolation)
Complexity Low High
Primary Use Case Local development/Testing Production CI/CD pipelines
Resource Control Docker daemon limits Kubernetes ResourceQuotas/Limits

The local Docker runner, as demonstrated in the bash script above, is highly effective for individual developers. By mounting /var/run/docker.sock, the runner can spawn sibling containers to execute jobs. However, this approach lacks the robust isolation and scaling capabilities provided by the Kubernetes executor. The Kubernetes executor, deployed via the Helm chart, is the standard for enterprise environments because it treats each job as a first-class Kubernetes citizen, leveraging the full power of the orchestrator to manage lifecycle, networking, and storage.

Conclusion

The deployment of GitLab Runner within a Rancher-managed ecosystem requires a sophisticated understanding of both Kubernetes orchestration and the specific idiosyncrasies of container runtimes. Successful implementation hinges on the precise configuration of the Helm values.yaml file, the rigorous application of kubeseal for secret security, and a clear understanding of the versioning relationship between the Runner application and its Helm chart.

The emergence of error code 143 in Rancher Desktop environments underscores the importance of testing CI/CD workflows in environments that closely mimic the final production target. While the Kubernetes executor provides the most scalable and isolated method for job execution, the complexities of local runtime interactions on Windows/WSL2 necessitate a cautious approach to local development environments. Ultimately, the transition from local Docker-based runners to a fully orchestrated Kubernetes-based runner deployment represents a shift toward more resilient, scalable, and secure automated software delivery pipelines.

Sources

  1. GitLab Runner in Kubernetes
  2. gitlab runner on wsl with rancher desktop not working #6503
  3. Install GitLab Runner
  4. GitLab Runner Helm chart

Related Posts