Secure Authentication via SSH Private Keys in GitLab

The implementation of Secure Shell (SSH) keys within the GitLab ecosystem represents the gold standard for authenticating identity and ensuring the integrity of data transfers between a local workstation or CI/CD runner and the GitLab server. By utilizing asymmetric cryptography—a system based on a mathematically linked pair of keys—users can push and pull code without the repetitive and potentially insecure requirement of entering a username and password for every single Git interaction. This mechanism is available across all service tiers, including Free, Premium, and Ultimate, and is supported across GitLab.com, GitLab Self-Managed, and GitLab Dedicated offerings.

The fundamental premise of this security architecture relies on the separation of the public and private keys. The public key is designed to be distributed openly and is uploaded to the GitLab account, where it acts as a lock. The private key, conversely, is the secret key that opens that lock and must be protected with absolute rigor on the local system. Because GitLab only ever possesses the public key, there is no risk of revealing confidential data by uploading it; however, the accidental upload of a private key would constitute a catastrophic security breach. Beyond simple authentication, the private key can be leveraged to sign commits, providing a cryptographically verifiable guarantee that the code was authored by the legitimate owner of the key, which can then be verified by anyone possessing the corresponding public key.

To facilitate this communication, the environment must have the OpenSSH client installed. This client is pre-installed on modern GNU/Linux distributions, macOS, and Windows 10. A minimum version of SSH 6.5 is required to ensure compatibility and security. The process of establishing this connection involves three primary phases: the generation of the key pair on the local system, the registration of the public key within the GitLab user profile, and the verification of the established connection.

SSH Key Generation and Management

The creation of an SSH key pair is the first step in establishing a secure tunnel to GitLab. Depending on the version of OpenSSH and the desired security level, different algorithms and formats are utilized.

For users running OpenSSH versions between 6.5 and 7.8, it is recommended to save private RSA keys in a more secure OpenSSH format to mitigate vulnerabilities. This is achieved by executing the following command:

ssh-keygen -o -f ~/.ssh/id_rsa

Alternatively, a new RSA key with high-strength encryption can be generated using a 4096-bit key length and a custom comment for identification:

ssh-keygen -o -t rsa -b 4096 -C "<comment>"

For those utilizing OpenSSH 8.2 or later, the use of hardware security keys (SK) is supported, allowing for the generation of ED25519SK or ECDSASK keys. This requires the physical insertion of a hardware security key into the computer before running:

ssh-keygen -t [key_type]

The comment added during this process is stored within the .pub file, allowing the user to identify which physical device or machine the key belongs to. If a user needs to update the passphrase for an existing key to enhance security or recover from a lost password, the following command is used:

ssh-keygen -p -f /path/to/ssh_key

Once the key is generated, the user must enter the passphrase at the prompts to finalize the update.

Integrating Public Keys with GitLab Accounts

The public key serves as the unique identifier for a user when interacting with GitLab via SSH. Because these keys bind directly to an account, they must be unique; a single public key cannot be associated with multiple GitLab accounts.

To add a key to a GitLab account, the user must copy the contents of the public key file. While this can be done manually, scripts are often used to extract the text. For ED25519 keys, the filename is typically id_ed25519.pub, whereas for RSA keys, it is id_rsa.pub.

The process of addition follows these steps:

• Navigate to the user profile page in the GitLab web interface.
• Click on the "Add Public Key" button.
• Paste the contents of the public key into the "Key" text field.
• Click the "Add key" button to finalize the association.

During this process, GitLab performs a critical security check against a database of known compromised keys. If a key is found to be compromised, GitLab will block its addition because the associated private keys are already public knowledge and could be used by malicious actors to gain unauthorized access. This restriction is hard-coded and cannot be configured or bypassed by the user. If a key is blocked, the only recourse is to generate a completely new SSH key pair.

Leveraging the SSH Agent for Seamless Workflow

Manually entering a passphrase for every Git operation is inefficient. The ssh-agent is a background process that holds decrypted private keys in memory, allowing the user to authenticate without re-entering the passphrase for the duration of the agent's session.

To initialize the agent and load a key, the following sequence is typically employed:

First, check for an existing agent and listed loaded keys:

ssh-add -l

If no agent is running, it must be started:

eval $(ssh-agent)

Then, the specific private key is added to the agent:

ssh-add ~/.ssh/id_rsa

For ED25519 keys, the command would be:

ssh-add ~/.ssh/id_ed25519

Once the key is loaded, the user can verify the loaded keys again using:

ssh-add -l

It is important to note that the ssh-agent process is tied to the current session. Logging out or rebooting the machine will terminate the process and clear the loaded keys. While this provides a layer of security, users must be aware that any process running on a machine where the ssh-agent is active may potentially leverage the loaded key.

Managing SSH Private Keys in GitLab CI/CD Pipelines

In the context of Automated Continuous Integration and Continuous Deployment (CI/CD), managing private keys requires specialized handling to avoid exposing secrets in job logs. There are two primary methods for injecting an SSH private key into a pipeline: using a file-type variable and using a regular variable.

File Type CI/CD Variables

The preferred method for handling SSH keys in CI/CD is the file-type variable. This is necessary because SSH keys contain whitespace characters and specific formatting (such as newlines) that would be corrupted by "Masked" or "Masked and hidden" regular variables, as these settings do not support whitespace.

To configure a file-type variable:

• Set the Visibility to "Visible".
• In the Key text box, define the variable name, such as SSH_PRIVATE_KEY.
• In the Value text box, paste the full content of the private key.
• Ensure the value ends with a newline (LF character) by pressing Enter at the end of the last line before saving.

A critical security warning: because the visibility is set to "Visible" and the variable is not masked, commands such as cat or tee must never be run on this variable in the .gitlab-ci.yml file, as this would print the private key directly into the job logs.

Pipeline Implementation and Execution

Once the variable is set, the private key must be loaded into the job's environment. This is done by running the ssh-agent within the job script:

bash eval $(ssh-agent -s) ssh-add <(echo "$SSH_PRIVATE_KEY")

The ssh-add - command is specifically used to prevent the value of the $SSH_PRIVATE_KEY variable from appearing in the job log, although it could still be exposed if debug logging is enabled.

If the pipeline needs to access a private GitLab repository, the corresponding public key must be added to that project as a "deploy key". This ensures the runner has the necessary permissions to clone the repository.

SSH Configuration for GitLab Runners and Shell Executors

The method of managing keys varies depending on the executor used by the GitLab Runner. When using a Docker executor, keys are usually passed via variables. However, when using the Shell executor, the process is more direct as the runner interacts directly with the host machine's operating system.

On a machine running the Shell executor, keys can be generated directly on the runner's host. The process is as follows:

• Log into the server running the jobs.
• Switch to the gitlab-runner user:

sudo su - gitlab-runner

• Generate a new SSH key pair. In this specific scenario, the key must be generated without a passphrase; otherwise, the before_script section of the pipeline will hang while waiting for manual passphrase input, causing the job to fail.
• Add the resulting public key to the target servers (usually located in ~/.ssh/authorized_keys) or as a deploy key in GitLab.
• Manually sign in to the remote server once to accept the SSH fingerprint:

ssh example.com

For repositories hosted on GitLab.com, the connection string is [email protected].

Advanced SSH Configuration and Server-Side Logic

For complex environments, users may need to manage multiple SSH identities or handle specific server configurations. The ~/.ssh/config file allows for the definition of host-specific settings to avoid conflicts.

An example configuration for different hosts:

```text
Host gitlab.com
PreferredAuthentications publickey
IdentityFile ~/.ssh/gitlabcomrsa

Host gitlab.company.com
PreferredAuthentications publickey
IdentityFile ~/.ssh/examplecomrsa
```

These settings ensure that the correct private key is mapped to the correct GitLab instance.

GitLab Server-Side SSH Daemon Logic

On the server side, GitLab integrates with the system's SSH daemon. It designates a specific user—typically named git—to handle all incoming SSH access requests. When a user connects, GitLab identifies them by their public key rather than a traditional username. This means all SSH operations are executed as the git user.

It is strongly discouraged to modify the SSH configuration for the git user to include a private SSH key for authentication requests. This practice creates significant security risks. GitLab actively checks for this condition during system audits. If a custom configuration is detected, the gitlab-rake gitlab:check command will report a failure:

Git user has default SSH configuration? ... no

To remediate this and return to a secure state, the custom keys should be backed up and removed:

bash mkdir ~/gitlab-check-backup-1504540051 sudo mv /var/lib/git/.ssh/id_rsa ~/gitlab-check-backup-1504540051 sudo mv /var/lib/git/.ssh/id_rsa.pub ~/gitlab-check-backup-1504540051

Security Analysis and Best Practices

The use of SSH keys fundamentally shifts the security burden from "something you know" (a password) to "something you have" (a private key). However, this introduces new risks that must be analyzed.

The primary risk associated with ssh-agent is the persistence of the decrypted key in memory. If a local machine is compromised, an attacker could potentially use the active agent to authenticate as the user. To mitigate this, users should avoid keeping agents running indefinitely and should utilize passphrases for their private keys.

Furthermore, the risk of Man-in-the-Middle (MITM) attacks exists during the initial connection to a server. It is a mandatory best practice to verify the private server's own public key (the host key) before trusting the connection.

In CI/CD environments, the "Visible" setting for file-type variables is a necessary evil due to the technical limitations of masked variables regarding whitespace. To secure this, users must strictly avoid any logging commands that output the contents of the variable. The use of deploy keys is also superior to using a personal user key in a pipeline, as deploy keys can be scoped to a specific project and revoked without affecting the user's global account access.

Summary of Technical Requirements and Specifications

The following table outlines the technical requirements for implementing SSH authentication within GitLab.

Conclusion

The implementation of SSH private keys in GitLab is a multifaceted process that balances convenience with high-level security. By moving away from password-based authentication, users gain a more robust and scalable method of interacting with their source code. The strength of this system lies in the asymmetric nature of the keys, provided that the private key remains confidential and the public key is uniquely mapped to a single GitLab account.

For developers, the use of ssh-agent is essential for productivity, but it must be managed with an understanding of the security implications of memory-resident keys. In the realm of DevOps and CI/CD, the transition to file-type variables and the strategic use of deploy keys ensure that automation does not come at the cost of security. The strict prohibition of compromised keys by GitLab further reinforces the ecosystem's integrity, forcing users toward modern, secure key generation standards. Ultimately, the correct configuration of the local environment, the adherence to non-masking variable protocols in pipelines, and the maintenance of a clean, default SSH configuration on the server side constitute the pillars of a secure GitLab deployment.

Sources

1. Use SSH keys with GitLab
2. SSH keys in CI/CD jobs
3. GitLab SSH Configuration (JP)
4. Version Control with Git and GitLab - Imperial College London