Secure Authentication Orchestration for GitLab Runner via SSH Key Injection and Remote Execution

The integration of Secure Shell (SSH) protocols within the GitLab Runner ecosystem represents a critical junction between automated continuous integration/continuous deployment (CI/CD) pipelines and secure infrastructure management. In a modern DevOps lifecycle, a GitLab Runner acts as the computational engine that executes jobs defined in a .gitlab-ci.yml file. However, these jobs frequently require elevated privileges or access to isolated network segments—such as private repositories, internal package registries, or remote deployment targets—that are not reachable via standard unauthenticated protocols.

The implementation of SSH keys within this workflow is not merely a convenience but a security requirement to facilitate non-interactive, cryptographically verified communication. Whether a runner is utilizing the Shell executor to interact with local resources or the specialized SSH executor to command remote hosts, the management of identity files, agent forwarding, and host key verification determines the stability and security of the entire deployment pipeline. This technical analysis explores the multifaceted methods of injecting SSH credentials, the architectural nuances of the SSH executor, and the rigorous security protocols necessary to prevent unauthorized access during automated build processes.

Use Cases for SSH Integration in CI/CD Pipelines

The requirement for SSH keys within a GitLab CI/CD environment arises from several distinct operational needs. Without a method to authenticate as a known entity, the runner remains an isolated agent, unable to fulfill the complex dependencies of modern software architectures.

The primary drivers for SSH implementation include:

Accessing Private Repositories and Submodules: When a project contains dependencies hosted in separate, private GitLab repositories, the runner must authenticate to pull these submodules. This is particularly common in PHP projects using composer.json or JavaScript projects using package.json, where internal packages are not public.
Private Package Management: Automated build processes often need to download private packages from internal registries using tools like Bundler. SSH provides the secure tunnel required to fetch these assets.
Remote Application Deployment: A fundamental CI/CD goal is to move code from a build environment to a production or staging server. SSH allows the runner to execute deployment scripts or use rsync to synchronize files to a remote host.
Remote Command Execution: The SSH executor specifically allows for the execution of build instructions on a remote machine, treating that machine as the execution environment itself.
Document Generation and Storage: In workflows where a CI job produces artifacts, such as converting Markdown to PDF, the resulting files may need to be pushed back to a repository or an external server via SSH-based transfers.

Architecting SSH Key Generation and Identity Management

Establishing a secure pipeline begins with the creation of a robust identity. SSH keys consist of a public component, which is distributed to the destination, and a private component, which must be guarded with extreme rigor.

Generating the Key Pair

To initiate the process, an SSH key pair should be generated on the machine where the GitLab Runner is installed or within a secure local environment. The industry standard for modern, high-security environments is the ed25519 algorithm, which offers superior performance and security compared to legacy RSA keys.

The command to generate this specific key type is:

bash ssh-keygen -t ed25519 -C "[email protected]"

During this process, several critical decisions must be made:

File Location: The user is typically prompted to accept the default location, which is ~/.ssh/id_ed25519.
Passphrase Security: While automated runners often require non-interactive access, adding a passphrase provides an additional layer of security. However, in a fully automated CI/CD context, managing this passphrase requires specific handling within the SSH agent.
Resulting Artifacts: The process yields two distinct files:
- ~/.ssh/id_ed25519: The private key, which must never be shared or exposed.
- ~/.ssh/id_ed25519.pub: The public key, which is safe to distribute.

If a specific name is required for organizational purposes, such as distinguishing between different GitLab instances, the -f flag can be used:

bash ssh-keygen -t ed25519 -C "christophe@my_self_hosted_gitlab" -f ~/.ssh/id_ed25519_my_self_hosted_gitlab

Distributing the Public Key

Once the pair is generated, the public key must be registered with the target destination to allow authentication.

For GitLab-specific authentication:
1. Log in to the GitLab account.
2. Access Profile Settings.
3. Navigate to the SSH Keys section.
4. Paste the content of the .pub file into the "Key" field.
5. Assign a descriptive name and an optional expiration date.

For remote server authentication:
The public key must be appended to the ~/.ssh/authorized_keys file on the destination server to grant the runner permission to log in.

Injecting Credentials via GitLab CI/CD Variables

GitLab does not provide a built-in mechanism for managing SSH keys directly within the runner's execution environment. Instead, the most widely supported and secure method is to inject the private key into the build environment using CI/CD variables. This method is compatible with both Docker and Shell executors.

Secure Variable Configuration

To prevent the exposure of sensitive credentials, the private key should be stored as a "File" type variable in GitLab.

SSHPRIVATEKEY: This variable stores the content of the private key.
SSHPUBLICKEY: This variable stores the content of the public key.

These variables can be configured at the Project level or the Group level. Utilizing the Group level allows for inheritance, where multiple projects within a group can share the same authentication credentials, simplifying management across a microservices architecture.

Implementation in the .gitlab-ci.yml File

To utilize these variables during a job, the .gitlab-ci.yml file must be configured to prepare the environment. The runner must create a secure directory, set the correct permissions, and load the key into an ssh-agent.

The following pattern is used in the before_script section:

yaml before_script: - eval $(ssh-agent -s) - mkdir -p ~/.ssh - chmod 700 ~/.ssh - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_ed25519 - chmod 600 ~/.ssh/id_ed25519

The impact of these commands is significant:
- eval $(ssh-agent -s): Starts the agent and sets the necessary environment variables.
- mkdir -p ~/.ssh: Ensures the directory exists, preventing job failure.
- chmod 700 ~/.ssh: Restricts directory access to the owner only, satisfying SSH security requirements.
- echo "$SSH_PRIVATE_KEY" ...: Reconstructs the private key file from the CI/CD variable. The tr -d '\r' command is a critical safeguard against Windows-style line endings that can corrupt key files.
- chmod 600 ~/.ssh/id_ed25519: Sets the strict permissions required for private keys; without this, the SSH client will reject the key as being too "open."

The SSH Executor: Remote Build Execution

The GitLab Runner's SSH executor is a specialized mode that allows the runner to execute build jobs on a remote machine by connecting via SSH. This is distinct from the Shell executor, which runs commands on the local machine where the runner is installed.

Configuration of the SSH Executor

To utilize this executor, the config.toml file of the GitLab Runner must be specifically configured. Unlike other executors, the SSH executor requires explicit parameters for the destination host.

Example configuration in config.toml:

toml [[runners]] executor = "ssh" [runners.ssh] host = "example.com" port = "22" user = "root" password = "password" identity_file = "/path/to/identity/file"

Important considerations for the SSH executor include:

Authentication Methods: The executor can authenticate using a plain text password or an identity_file. Using an identity_file is the preferred professional standard for security.
Identity File Explicitly: A critical technical nuance is that the GitLab Runner does not implicitly read identity files from the standard locations like /home/user/.ssh/id_rsa. The identity_file path must be explicitly defined in the configuration.
Script Limitations: The SSH executor only supports scripts generated in Bash.
Caching Support: It is important to note that the caching feature is not supported when using the SSH executor.

Host Key Verification and StrictHostKeyChecking

When a runner connects to a remote host for the first time, the SSH protocol initiates a "host key verification" process to prevent Man-in-the-Middle (MITM) attacks. This requires the host's fingerprint to be present in the ~/.ssh/known_hosts file.

In a CI/CD context, this can cause jobs to hang indefinitely as the runner waits for a user to confirm the host's authenticity. To manage this, there are two primary approaches:

Manual Fingerprint Addition: Adding the GitLab server's SSH fingerprint to the ~/.ssh/known_hosts file within the job script.
Disabling Strict Checking: In the config.toml for the SSH executor, the setting disable_strict_host_key_checking can be manipulated.

Configuration Setting	Default Value (GitLab 15.0+)	Impact
`disable_strict_host_key_checking`	`false`	When `false`, host key checking is required, increasing security but requiring manual fingerprint management.
`disable_strict_host_key_checking`	`true`	When `true`, the runner ignores host key mismatches, which simplifies automation but exposes the pipeline to MITM attacks.

Filesystem and Pathing Nuances

Understanding the location of files within the runner's environment is vital for successful job execution, especially when debugging pathing issues or managing artifacts.

The Build Directory Structure

When a job is executed, the project source is checked out into a specific directory structure on the runner. This path follows a predictable pattern:

~/builds/<short-token>/<concurrent-id>/<namespace>/<project-name>

Where:
- <short-token>: A shortened version of the runner's token (first 8 characters).
- <concurrent-id>: A unique number identifying the local job ID on the specific runner.
- <namespace>: The GitLab namespace (group or user) where the project resides.
- <project-name>: The specific name of the repository.

If an administrator needs to change this default behavior, they can specify the builds_dir option under the [[runners]] section within the config.toml file.

Artifact Uploading Requirements

If a job is executed via the SSH executor and the intention is to upload job artifacts back to GitLab, the gitlab-runner package must be installed on the remote host being connected to. This is because the remote host is the one responsible for the actual execution and subsequent communication with the GitLab API to upload the files.

Security Best Practices and Risk Mitigation

The intersection of automation and SSH introduces significant security surface areas. Failure to implement these protocols correctly can lead to credential leakage or unauthorized server access.

Avoid Personal Key Reuse: Never use a personal SSH key that is tied to an individual user's account for automated CI/CD jobs. If the personal account is compromised or the user leaves the organization, the pipeline is broken or exposed.
Regular Key Rotation: Implement a policy for rotating SSH keys used in CI/CD variables to minimize the window of opportunity for an attacker using a leaked key.
Principle of Least Privilege: The user account used by the GitLab Runner (e.g., the user in the SSH configuration) should only have the permissions necessary to perform the required tasks. For example, if the runner only needs to upload files via rsync, it should not have root access.
Secure Variable Storage: Use GitLab's "Masked" variable feature if possible, although standard SSH private keys often contain characters that prevent masking, making the use of "File" type variables and strict directory permissions the primary defense.
SSH Fingerprint Management: Rather than disabling StrictHostKeyChecking, the more secure method is to include the command ssh-keyscan <hostname> >> ~/.ssh/known_hosts in the before_script to securely register the host.

Technical Analysis of Execution Flows

The operational success of an SSH-enabled GitLab Runner depends on the synchronization of three distinct layers: the GitLab Server (the orchestrator), the Runner Machine (the executor), and the Target Host (the destination).

In a standard SSH-based deployment flow, the orchestration follows this sequence:
1. The GitLab Server triggers a job and sends the instructions to the Runner.
2. The Runner initializes the environment, pulling the SSH_PRIVATE_KEY from the encrypted GitLab database.
3. The Runner's local agent (or the injected environment) prepares the identity file with 600 permissions.
4. The Runner establishes an encrypted tunnel to the Target Host.
5. The Target Host validates the public key against its authorized_keys.
6. Commands are executed, and if configured, artifacts are streamed back to the GitLab Server.

Disruptions in this flow typically occur at the permission layer (e.g., chmod 700 not being applied to ~/.ssh) or the network layer (e.g., host key mismatch). By treating the SSH configuration as a code-managed entity within the .gitlab-ci.yml and config.toml files, engineers can ensure a repeatable and secure deployment lifecycle.