The integration of Secure Shell (SSH) within the GitLab Continuous Integration (CI) framework represents a critical bridge between the automated build environment and the physical or virtual infrastructure where applications reside. While modern software development has pivoted heavily toward container orchestration and serverless architectures, the requirement to deploy code to traditional Linux servers remains a prevalent necessity. This process involves transforming a GitLab Runner—often a transient Docker container—into a secure client capable of authenticating with a remote server to execute commands, transfer files, and manage application states. Achieving this requires a precise orchestration of cryptographic keys, environment variables, and shell configurations to ensure that the automation is both seamless and secure.
Architectural Foundations of SSH in GitLab CI
The primary objective of using SSH within a GitLab CI pipeline is to facilitate "auto deploy" capabilities. This allows a developer to move beyond simple auto-compiling and auto-testing, ensuring that once a codebase passes its quality gates, it is automatically uploaded to a production or staging server. This mechanism mimics the behavior of Platform-as-a-Service (PaaS) providers like Heroku, where the push of code triggers an automated deployment sequence.
The technical execution of this process relies on the SSH protocol, which extends beyond simple remote shell access. In the context of GitLab CI, SSH is utilized for two primary functions:
- Remote Command Execution: The ability to trigger pre-build or post-build scripts on the remote server, such as restarting a service, clearing a cache, or migrating a database.
- Remote Folder Mounting: Utilizing tools like SSHFS to interact with remote directories as if they were local, facilitating the movement of build artifacts.
Remote Server Preparation and Security Hardening
Before a GitLab CI pipeline can successfully communicate with a remote server, the target environment must be specifically configured to accept connections from the CI runner. In a typical Linux environment, such as CentOS 7, several administrative steps are mandatory to establish a secure handshake.
The first step is the creation of a dedicated deployment user. Rather than using a high-privilege account like root, it is recommended to create a specific user, such as ci, to limit the blast radius of any potential security breach. This user acts as the entry point for the automated pipeline.
Once the user is created, the server must be configured to recognize the public key of the GitLab Runner. The SSH daemon (sshd) relies on the authorized_keys file to validate incoming connections. This file is located at /home/ci/.ssh/authorized_keys and contains a list of public keys, one per line, that are permitted to log in as that specific user.
After modifying the authorized_keys file, the SSH daemon must be restarted to ensure all configurations are active. This is performed using the following command:
sudo systemctl restart sshd
The security of this setup relies on the correct permissioning of the .ssh directory and the authorized_keys file. If permissions are too permissive, the SSH daemon may reject the key for security reasons.
SSH Key Generation and Management
The authentication mechanism is based on asymmetric cryptography, involving a public key and a private key. To facilitate the connection from the GitLab CI Docker container to the server, a key pair must be generated. This generation can be performed on any spare machine, such as a local workstation or a virtual machine.
The generation process is initiated with the following command:
ssh-keygen
By default, this command generates two files in the /home/<username>/.ssh/ directory:
id_rsa.pub: The public key, which is uploaded to the remote server'sauthorized_keysfile.id_rsa: The private key, which must be kept secret and provided to the GitLab CI environment.
It is imperative that no passphrase is added to the SSH key during generation. If a passphrase is used, the before_script in the CI pipeline will prompt for manual input, which will cause the automated job to hang and eventually fail, as there is no interactive terminal available in a CI environment.
Configuring GitLab CI/CD Variables
Because the private key is a sensitive piece of information, it cannot be committed to the git repository. Instead, it must be stored within GitLab's CI/CD variables settings, found under the project's Settings menu in the CI/CD section.
There are two primary methods for storing these keys:
File Type Variables
The preferred method is setting the variable type to "file". When a variable is designated as a file, GitLab stores the content in a temporary file on the runner and provides the path to that file in the environment variable. This method is superior because it preserves multiline formatting, which is essential for the structural integrity of an RSA private key.
Regular Variable Type
While possible, using a regular variable is generally discouraged. Regular variables are treated as strings, and if the key is not handled correctly, it can lead to formatting errors. Furthermore, the private key cannot be "masked" in the GitLab UI because it does not meet the specific character requirements for masking, meaning it will not be hidden in the logs if printed.
To mitigate security risks, these variables should be restricted to specific branches or tags. This prevents a developer from creating a feature branch and modifying the .gitlab-ci.yml file to output the private key in plain text to the job logs.
Implementation within the .gitlab-ci.yml Pipeline
The transition from a stored variable to a functioning SSH connection requires specific steps within the .gitlab-ci.yml file, particularly in the before_script section.
Environment Setup and Directory Creation
A common failure point in GitLab CI pipelines is the absence of the .ssh directory within the transient Docker container. If the pipeline attempts to write a key to a directory that does not exist, it will trigger a "No such file or directory" error. To prevent this, the following commands must be executed:
mkdir -p ~/.ssh
chmod 700 ~/.ssh
These commands ensure the directory exists and has the strict permissions required by SSH.
Key Deployment and Formatting
Once the directory is created, the private key stored in the variable must be placed into the environment. For example, if using a regular variable:
cat $SSH_PRIVATE_KEY > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
A critical technical detail often overlooked is the requirement for a newline character at the end of the private key. If the key is copied into the GitLab CI variable without a trailing blank line, the ssh-add command or the SSH client may throw an "error in libcrypto". To resolve this, users must manually ensure there is a return/enter character at the end of the key in the GitLab variable settings.
Resolving Host Key Verification Failures
Even after the keys are correctly configured, the SSH connection will often fail due to the "Host key verification failed" error. This happens because the remote server's public key is not present in the runner's known_hosts file, and the runner cannot verify the identity of the server it is connecting to.
There are three primary methods to resolve this issue:
The Manual Known Hosts Method
If the server's public key is known, it can be stored as another CI/CD variable (e.g., SSH_KNOWN_HOSTS). This variable is then appended to the known_hosts file during the before_script:
echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
chmod 644 ~/.ssh/known_hosts
The Key-Scan Method
The ssh-keyscan utility can be used to dynamically retrieve the public key of the remote host and add it to the known_hosts file. This is a flexible approach for environments where host keys might change.
ssh-keyscan example.com >> ~/.ssh/known_hosts
chmod 644 ~/.ssh/known_hosts
Disabling Strict Host Key Checking
For environments where security is less critical than connectivity, the StrictHostKeyChecking option can be disabled. However, this is generally not recommended as it exposes the connection to man-in-the-middle (MITM) attacks.
ssh -o StrictHostKeyChecking=no "[email protected]"
GitLab Runner SSH Executor
In addition to using SSH commands within a Docker-based pipeline, GitLab provides a specific "SSH Executor". This is a different architectural approach where the GitLab Runner itself connects to an external server to run the entire build process there, rather than running the build in a container and then deploying to the server.
SSH Executor Specifications
The SSH executor is available across all tiers (Free, Premium, and Ultimate) and is supported on GitLab.com, Self-Managed, and Dedicated offerings. However, it is currently in maintenance mode, meaning it receives critical security updates but no new features.
| Feature | SSH Executor Support |
|---|---|
| Script Language | Bash only |
| Caching | Not supported |
| Connection Method | External server via SSH |
| Status | Maintenance Mode |
Configuration of the SSH Executor
To implement the SSH executor, the config.toml file of the runner must be configured as follows:
toml
[[runners]]
executor = "ssh"
[runners.ssh]
host = "example.com"
port = "22"
user = "root"
password = "password"
identity_file = "/path/to/identity/file"
Authentication can be achieved through a password, an identity file, or both. It is important to note that the GitLab Runner does not automatically read identity files from the default ~/.ssh/id_rsa location; the path must be explicitly defined in the configuration.
Comparative Analysis of Deployment Methods
When deciding between using an SSH-based deployment script within a Docker executor and using the SSH executor, the choice depends on the desired level of isolation and the build requirements.
| Method | Docker Executor + SSH Scripts | SSH Executor |
|---|---|---|
| Isolation | High (Containerized) | Low (Runs directly on host) |
| Caching | Supported | Not supported |
| Flexibility | High (Can use any image) | Limited (Bash only) |
| Setup Complexity | Medium (Requires key config) | Low (Configured in Runner) |
| Security | High (Ephemeral environment) | Medium (Persistent host access) |
Advanced Troubleshooting and Operational Best Practices
To ensure a robust deployment pipeline, several operational safeguards should be implemented.
One highly recommended security measure is to restrict SSH access to specific static IP addresses. By locking down the server's firewall to only allow connections from the GitLab Runner's IP range, the attack surface is significantly reduced.
When encountering the libcrypto error during key loading, the first point of inspection should always be the trailing newline in the CI variable. This is a frequent point of failure due to how different operating systems handle line endings.
Finally, for those using Debian-based images in their Docker executors, the ssh-agent must be installed and configured in the before_script to manage the keys properly. The ssh-agent allows the private key to be decrypted once and used for multiple subsequent SSH commands without needing to re-specify the key file.
Conclusion
The successful implementation of SSH within GitLab CI requires a meticulous approach to both server-side configuration and pipeline orchestration. By establishing a dedicated ci user, carefully managing asymmetric key pairs through "file" type CI/CD variables, and resolving host verification through ssh-keyscan or known_hosts management, organizations can achieve a highly efficient and secure automated deployment workflow. While the SSH executor provides a simpler path for some, the combination of Docker executors and SSH scripting offers the most flexibility and isolation, provided the practitioner adheres to the strict permissioning and formatting requirements of the SSH protocol.