Secure Remote Command Execution via GitLab CI/CD SSH Integration

Establishing a secure, automated communication channel between a GitLab CI/CD pipeline and a remote server is a fundamental requirement for modern DevOps workflows. Because GitLab does not provide built-in, native management of SSH keys within the build environment where the GitLab Runner operates, engineers must implement a manual injection strategy. This process ensures that the ephemeral environment created by the Runner—whether it is a Docker container or a shell executor—possesses the necessary cryptographic credentials to authenticate with a target server without requiring human intervention.

The application of SSH keys within GitLab CI/CD is versatile and critical for several high-impact scenarios. Primarily, it allows for the checkout of internal submodules and the downloading of private packages via package managers such as Bundler, which would otherwise fail due to lack of authentication. Furthermore, it is the primary mechanism for deploying applications to proprietary servers or platforms like Heroku, executing remote shell commands, and utilizing Rsync to synchronize files from the build environment to a production or staging server.

For an implementation to be successful and secure, it must adhere to strict cryptographic standards. This involves generating a unique SSH key pair specifically for the automation task, avoiding the reuse of personal keys, and implementing a regular rotation schedule. The failure to rotate keys or the use of a single key across multiple environments increases the blast radius of a potential credential leak, potentially granting an attacker unauthorized access to the entire infrastructure.

SSH Key Implementation Strategies

There are two primary methods for injecting SSH keys into a GitLab CI/CD pipeline: using file-type variables and using regular variables. The choice between these two significantly impacts how the key is handled by the runner and how it is formatted in the .gitlab-ci.yml configuration.

File Type CI/CD Variables

The most widely supported and recommended method is the use of a file-type CI/CD variable. In this configuration, GitLab stores the key and automatically creates a temporary file on the runner's disk, providing the path to that file via an environment variable.

• Visibility Settings: The visibility must be set to Visible. This is mandatory because SSH keys contain whitespace characters; Masked or Masked and hidden variables do not support whitespace, which would corrupt the key format.
• Formatting Requirements: The value pasted into the Key text box must end with a newline (LF character). Users must press Enter or Return at the end of the last line before saving to ensure the key is read correctly by the SSH client.
• Security Risks: Because these variables are not masked, running commands such as cat or tee on the variable will expose the private key in the job logs.
• Management Advantage: File-type variables are preferred because they preserve multiline formatting, which drastically reduces the risk of formatting-related errors during the injection process.

Regular CI/CD Variables

Alternatively, users can store the SSH key as a regular variable. This method requires the user to manually handle the creation of the key file within the script section of the pipeline.

• Manual File Creation: The user must use a command such as echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa to write the variable content to a file.
• Encoding Considerations: Some users employ Base64 encoding for the variable to avoid whitespace issues, using a command like echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa. However, this adds an extra layer of complexity and is often unnecessary if the variable is handled correctly.
• Permission Management: When creating the file manually, it is critical to set the correct permissions using chmod 600, as SSH clients will reject private keys that are too open.

Technical Configuration and Workflow

The integration of SSH keys requires a specific sequence of operations within the .gitlab-ci.yml file to transition from a blank slate to an authenticated session.

Key Generation and Deployment

The process begins outside of GitLab. A new SSH key pair must be generated without a passphrase. If a passphrase is added, the before_script will hang indefinitely as it waits for a user to enter the passphrase, which is impossible in a non-interactive CI/CD environment.

The public key must then be deployed to the target server. This is typically done by appending the public key to the ~/.ssh/authorized_keys file of the user account that the pipeline will use to log in. For example, if the pipeline connects as a user named deployer, the public key must be placed in /home/deployer/.ssh/authorized_keys. If the connection is made as root, it must be in /root/.ssh/authorized_keys.

The Runner Environment Setup

Depending on the executor, the setup varies. For Docker executors, the environment is completely isolated, meaning all necessary tools must be installed during the job's execution.

• Package Installation: In Alpine-based images, the openssh-client must be installed via apk add openssh-client. In Debian-based images, the ssh-agent must be installed.
• SSH Agent Execution: The ssh-agent should be started to load the private key into memory, which simplifies the authentication process for subsequent commands.
• Key Placement: The private key is moved to the ~/.ssh/ directory, often named id_rsa, and the permissions are strictly limited.

Host Key Verification and Security

A common failure point in CI/CD pipelines is the "Host key verification failed" error. This happens because the remote server's fingerprint is not known to the runner.

• The Risks of ssh-keyscan: Running ssh-keyscan directly inside a CI/CD job is a security risk as it leaves the pipeline vulnerable to machine-in-the-middle (MITM) attacks.
• The Secure Alternative: The recommended approach is to use a file-type CI/CD variable named SSH_KNOWN_HOSTS.
• Pre-calculating Fingerprints: The administrator should run ssh-keyscan example.com or ssh-keyscan 10.0.2.2 from a trusted network to obtain the host keys.
• Implementation in Pipeline: The before_script should copy this variable to the known hosts file:
  cp "$SSH_KNOWN_HOSTS" ~/.ssh/known_hosts
chmod 644 ~/.ssh/known_hosts

By pre-defining the host keys, the pipeline is protected against unexpected changes in the server's identity. If the host keys change suddenly, the job will fail, signaling a potential security breach or a critical server change.

Execution of Remote Commands

A frequent error encountered by users is the failure of subsequent commands to execute after an SSH connection is established. This typically occurs when commands are listed as separate lines in the script block rather than as part of the SSH command itself.

Command Stringing

When a user executes a command like:
ssh -i "ssh_key.pem" centos@<ip>
pwd
ls -l /etc/gitlab

The pwd and ls commands are executed on the GitLab Runner's local environment, not the remote server. To execute commands on the remote server, they must be passed as a single string argument to the SSH command.

• Correct Syntax: The commands should be joined using && to ensure that the second command only runs if the first one succeeds.
• Example Execution: ssh -i "ssh_key.pem" -t -t -o StrictHostKeyChecking=no centos@<ip> "pwd && ls -la /etc/gitlab"

Handling Complex Deployments

For more complex tasks, such as deploying code from a repository into a remote server, the pipeline must navigate to the working directory and interact with Git.

• Command Chain: ssh $SSH_USER@$SSH_HOST "cd $WORK_DIR && git checkout $PRELIVE_BRANCH && git pull && exit"
• Error Analysis: If a "Permission denied" error occurs, it is often because the public key was added to the wrong user's authorized_keys file (e.g., added to root but the pipeline is connecting as deployer).
• Libcrypto Errors: Errors such as Load key "/root/.ssh/id_rsa": error in libcrypto often indicate a formatting issue with the private key or an attempt to use a key that was not correctly decoded from Base64.

Comparison of SSH Variable Types

Troubleshooting and Common Failures

The following table outlines common failures when attempting to establish SSH connections within GitLab CI/CD and their respective resolutions.

Detailed Implementation Example

To ensure a successful deployment, the .gitlab-ci.yml should be structured to handle the environment setup in the before_script and the cleanup in the after_script.

The before_script must perform the following sequence:
1. Update the package manager: apk update.
2. Install the SSH client: apk add openssh-client.
3. Create the .ssh directory if it doesn't exist.
4. Inject the private key: echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa (if using Base64).
5. Set the private key permissions: chmod 600 ~/.ssh/id_rsa.
6. Inject the known hosts: ssh-keyscan -H $SSH_HOST > ~/.ssh/known_hosts.

The script section then executes the remote command:
ssh $SSH_USER@$SSH_HOST "cd $WORK_DIR && git pull"

Finally, the after_script ensures that the credentials do not persist in the environment:
rm -rf ~/.ssh

Conclusion

Integrating SSH into GitLab CI/CD is a balancing act between automation and security. The most robust architecture relies on file-type variables to maintain the integrity of multiline private keys and the pre-calculation of host fingerprints via SSH_KNOWN_HOSTS to prevent man-in-the-middle attacks. The critical technical takeaway for developers is the distinction between local and remote execution; any command intended for the server must be encapsulated within the SSH call's argument string. By adhering to the principle of least privilege—using non-root users for deployment and rotating keys regularly—organizations can achieve a secure and scalable deployment pipeline that mitigates the risks associated with automated credential management.

Sources

1. Using SSH keys with GitLab CI/CD
2. ssh to ec2 instance through gitlab cicd and run commands
3. deploying code from gitlab repository into remote server