The mechanism by which GitLab Runner handles Transport Layer Security (TLS) certificates is a critical component of the CI/CD pipeline, especially when operating within enterprise environments utilizing self-signed certificates or internal Certificate Authorities (CA). At the center of this process is the CI_SERVER_TLS_CA_FILE variable, a predefined environment variable that points to the CA certificate used by the GitLab server. When this mechanism fails, the result is typically a catastrophic halt of the pipeline during the "Getting source from Git repository" stage, manifesting as certificate verification errors. This failure often stems from path duplication bugs, missing files, or the expiration of root certificates, such as the DST Root CA X3 event, which disrupts the chain of trust between the runner and the server.
The Mechanics of CISERVERTLSCAFILE and Certificate Trust
The CI_SERVER_TLS_CA_FILE is designed to provide the GitLab Runner with the necessary cryptographic material to verify the identity of the GitLab server. In a standard HTTPS connection, the client (the runner) verifies the server's certificate against a list of trusted root CAs. When a custom or self-signed certificate is used, the runner must be explicitly told where the trusted root resides.
The GitLab Runner implements this by exposing the path to the certificate via the CI_SERVER_TLS_CA_FILE environment variable. This allows tools within the build environment, such as curl or the git binary, to reference the specific certificate required for the session. For instance, a user can execute a command to retrieve a file using the server's CA cert by running:
curl --cacert "${CI_SERVER_TLS_CA_FILE}" ${URL} -o ${FILE}
This variable is essential for maintaining the chain of trust. If the file pointed to by this variable is missing or the path is incorrectly mapped, the git clone or git fetch operations will fail because the underlying Git process cannot verify the SSL certificate of the remote repository.
Analysis of Path Duplication and File System Failures
A recurring and severe issue identified in various runner configurations is the corruption of the file path associated with the CA certificate. This manifests as a "doubled" path, where the base directory of the build is prepended to the actual intended path, leading to a "No such file or directory" error.
In one documented instance, a build succeeded in its Maven phase but failed immediately after with the following error:
/bin/bash: line 20: /builds/vsimrti/eject/builds/vsimrti/eject.tmp/CI_SERVER_TLS_CA_FILE: No such file or directory
In this case, the path /builds/vsimrti/eject was repeated. This indicates a failure in how the runner environment resolves the temporary path for the TLS CA file. This is not merely a cosmetic error; it is a functional failure that prevents the shell from locating the certificate, resulting in an exit code 1.
Similar behavior is observed in Windows environments using the Shell executor. A specific failure case showed the path as:
CAfile: D:\Gitlab Runner\D:\Gitlab Runner\builds\d4db4ac7\0\xxx\xxx.tmp\GIT_SSL_CAINFO
The duplication of D:\Gitlab Runner\ creates an invalid filesystem path. Because the operating system cannot find this path, the Git process returns a fatal error stating "error setting certificate verify locations," which subsequently leads to an exit status 128. This is particularly prevalent in older versions of the multi-runner on Windows 10, where the internal logic for constructing the path to the .tmp certificate file fails to account for the root installation directory correctly.
Certificate Verification Failures in Shell Executors
The Shell executor is particularly susceptible to certificate issues because it relies on the host machine's environment and the specific version of Git installed on that host. When the CI_SERVER_TLS_CA_FILE is incorrectly mapped or the system's root certificates are outdated, the runner will fail to access the repository.
A common error seen in macOS (M1) and Linux environments is:
fatal: unable to access 'https://gitlab.mycompany.net/app/app.git/': error setting certificate verify locations: CAfile: /Users/ci/builds/nQMt-qB3/0/app/app.tmp/CI_SERVER_TLS_CA_FILE CApath: none
This indicates that while the runner is attempting to use the CI_SERVER_TLS_CA_FILE mechanism, the file does not actually exist at the specified temporary location. This is often a symptom of a version mismatch or a bug in the runner's ability to provision the temporary certificate file into the build directory before the Git operation commences.
The impact of this failure is total pipeline blockage. Even if the certificate is valid for other tools like Safari or curl on the same machine, the Git binary used by the GitLab Runner adheres to the path provided by the runner's configuration. If that path is broken, the clone operation fails, and the job is marked as failed.
Impact of Global Root Certificate Expirations
Beyond pathing errors, systemic failures can occur due to the expiration of global root certificates. A primary example is the expiration of the DST Root CA X3 certificate in September 2021. This event caused widespread failures in CI/CD pipelines that relied on Let's Encrypt certificates.
Users reported that pipelines working perfectly before September 29, 2021, suddenly began failing with:
unable to access 'https://gitlab-ci-token:[MASKED]@domainname/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
In these scenarios, the failure is not due to a missing CI_SERVER_TLS_CA_FILE but rather that the ca-certificates.crt file on the host system no longer trusts the chain provided by the server. Efforts to resolve this often involve an exhaustive list of unsuccessful attempts, such as:
- Restarting the Ubuntu server.
- Reconfiguring the GitLab service via
gitlab.rb. - Running
sudo update-ca-certificates. - Updating the system OpenSSL version (e.g., version 1.0.2g).
- Manually editing
ca-certificates.confto disable the outdated DST Root CA X3. - Manually adding
ISRG_Root_X1.crtto/etc/gitlab/trusted-certs/.
The persistence of this error, even after updating the system certificates, highlights that the GitLab Runner may be using a specific cached version of the certificate or a specific path that does not align with the system's updated trust store.
Configuration Strategies for Custom CA Certificates
To prevent the aforementioned failures, GitLab provides several official methods for managing TLS certificates. The choice of method depends on the executor being used (Docker vs. Shell) and the operating system.
Methods for Implementing Trust
The GitLab Runner can be configured to trust a server through two primary mechanisms:
- System Certificate Store: By default, the runner reads the system's existing certificate store. This is the most stable method if the host OS is kept up to date.
- Custom Certificate File: The
tls-ca-fileoption can be used during registration viagitlab-runner register --tls-ca-file=/pathor by adding the path directly to theconfig.tomlfile under the[[runners]]section.
For Docker executors, the most reliable way to ensure certificates are present is to use a pre_build_script to install the certificates into the container before the job starts.
Implementation for Ubuntu Docker Containers
For Ubuntu-based images, the following configuration in config.toml ensures the CA certificate is installed and updated:
toml
[[runners]]
name = "docker"
url = "https://example.com/"
token = "TOKEN"
executor = "docker"
pre_build_script = """
apt-get update -y > /dev/null
apt-get install -y ca-certificates > /dev/null
cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
update-ca-certificates --fresh > /dev/null
"""
Implementation for Alpine Docker Containers
For Alpine-based images, the process is similar but uses the apk package manager:
toml
[[runners]]
name = "docker"
url = "https://example.com/"
token = "TOKEN"
executor = "docker"
pre_build_script = """
apk update >/dev/null
apk add ca-certificates > /dev/null
rm -rf /var/cache/apk/*
cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
update-ca-certificates --fresh > /dev/null
"""
In both cases, the runner helper image installs the user-defined ca.crt file at startup. This file can be mapped to specific locations depending on the OS:
- Linux:
/etc/gitlab-runner/certs/ca.crt - Windows:
C:\GitLab-Runner\certs\ca.crt
Technical Comparison of Certificate Handling Methods
| Method | Mechanism | Best For | Potential Failure Point |
|---|---|---|---|
| System Store | OS Trust Store | Standard Public Certs | Root CA Expiration (e.g., DST Root CA X3) |
tls-ca-file |
Explicit path in config.toml |
Self-signed Certs | Path duplication bugs in Shell executor |
pre_build_script |
Dynamic installation in Docker | Ephemeral Containers | Network failure during apt-get update |
CI_SERVER_TLS_CA_FILE |
Env variable pointing to cert | Tools like curl |
Missing file in .tmp directory |
Root Cause Analysis of the "Doubled Path" Bug
The "doubled path" phenomenon is a critical failure in the GitLab Runner's internal path construction logic. When the runner prepares the environment, it attempts to create a temporary file containing the CA certificate and sets the CI_SERVER_TLS_CA_FILE variable to that path.
The logic failure occurs when the runner takes the current working directory (CWD) and appends the relative path to the temporary certificate. If the runner incorrectly assumes the path is relative when it is already absolute, or if it applies the CWD prefix twice, the resulting string becomes an invalid path.
For example, if the build directory is /builds/vsimrti/eject and the intended temporary file is in /builds/vsimrti/eject.tmp/CI_SERVER_TLS_CA_FILE, a logic error results in:
/builds/vsimrti/eject + /builds/vsimrti/eject.tmp/CI_SERVER_TLS_CA_FILE.
This is an infrastructure-level bug within the gitlab-runner binary itself. The only workaround for users experiencing this in Shell executors is to either update the runner to a version where this is patched or to manually bypass the CI_SERVER_TLS_CA_FILE by configuring Git to ignore SSL verification (which is a security risk) or by pointing Git to a static, non-temporary CA file on the host system.
Analysis of the "Exit Status 128" Failure
The exit status 128 encountered in Windows environments is the standard Git return code for a fatal error. In the context of GitLab Runner, this is almost always tied to the inability of the Git binary to initialize the SSL library because the CAfile path is invalid.
When the Git binary receives the command to clone a repository, it reads the GIT_SSL_CAINFO or the path provided by the runner. If the path is D:\Gitlab Runner\D:\Gitlab Runner\..., the Windows API returns a "system cannot find the path specified" error. Because the Git operation is the very first step of the pipeline ("Getting source from Git repository"), the failure prevents the creation of the .git directory. Consequently, any subsequent steps that attempt to interact with the repository (like Checking out fe9f705f as master) fail with the error:
fatal: Not a git repository (or any of the parent directories): .git
This creates a cascading failure where the environment is never fully initialized, making it impossible for the user to run diagnostic commands within the pipeline to debug the pathing issue.
Conclusion
The failures surrounding CI_SERVER_TLS_CA_FILE represent a intersection of environment configuration, software bugs, and global cryptographic shifts. The duplication of paths in both Linux and Windows environments suggests a systemic issue in how certain versions of GitLab Runner handle the construction of temporary certificate paths, transforming a necessary security feature into a point of failure.
The DST Root CA X3 expiration event further demonstrated that relying solely on the system's default certificate store is insufficient for enterprise-grade stability. The most robust solution for avoiding these issues is the implementation of a dual-layered trust strategy: utilizing the tls-ca-file configuration for the runner's own communication and implementing a pre_build_script in Docker executors to ensure that the environment's internal tools (Git, curl, etc.) possess a fresh and valid copy of the CA certificate. For Shell executors, ensuring that the host's Git version is current and that the runner is updated to a version that resolves the path-doubling bug is the only sustainable path to stability.