Remote Server Orchestration via GitLab CI and SSH

The integration of GitLab CI/CD with Secure Shell (SSH) protocols represents a fundamental architecture for automating the delivery of software from a version-controlled repository to a live production or staging environment. While modern containerization and orchestration tools like Kubernetes have shifted the paradigm toward immutable infrastructure, the requirement to execute commands or deploy files directly to a remote Linux virtual machine remains a critical necessity for many enterprise and hobbyist workflows. Utilizing GitLab CI/CD to manage SSH connections allows developers to transition from manual, error-prone deployments to a standardized, repeatable pipeline. This process involves the orchestration of a GitLab Runner, the secure management of cryptographic keys, and the configuration of a pipeline definition file that instructs the runner to authenticate and execute specific commands on a target host.

Infrastructure Requirements and Initial Setup

To establish a functional deployment pipeline using SSH, several core components must be present and correctly configured. The absence of any single component will result in a pipeline failure, typically manifesting as an authentication error or a connection timeout.

The foundational requirements include:

• A GitLab account: This provides the version control system and the CI/CD engine.
• A remote server: A Linux-based virtual machine is required. Examples include Azure Linux VMs or other cloud-based compute instances.
• A GitLab project: A repository must be initialized to house the source code and the pipeline configuration.
• SSH Key Pairs: A public and private key pair must be generated to facilitate passwordless authentication.
• A GitLab CI/CD pipeline: A defined sequence of jobs that execute the deployment logic.

When initializing a project for the first time, users may leverage GitLab templates to accelerate the process. For instance, selecting the Pages/Plain HTML template creates a baseline project structure that includes a README.md file and an initial .gitlab-ci.yml file, along with a public directory containing index.html and style.css. This structure provides a tangible set of files to deploy, serving as a proof-of-concept for the SSH connection.

The Role of the GitLab Runner

The GitLab Runner is the agent that executes the jobs defined in the .gitlab-ci.yml file. It acts as the execution environment where the SSH commands are initiated. There are two primary deployment models for runners:

• Self-managed Runners: These are installed by the user on their own infrastructure, providing full control over the environment and security.
• Shared Runners: These are maintained by GitLab and are available out-of-the-box. They require no configuration from the user and are ideal for standard deployments. GitLab provides a free tier offering 400 minutes per month for these runners.

The choice of runner impacts how the SSH environment is initialized. Since shared runners typically use Docker executors, the environment is ephemeral. This means that any SSH keys or configuration files created during a job are wiped once the job completes. Consequently, the pipeline must re-inject the SSH private key and re-configure the known_hosts file every time the job runs.

Advanced SSH Key Management

GitLab does not provide a built-in, centralized manager for SSH keys within the build environment. Instead, it provides a mechanism to inject these keys using CI/CD variables. This is essential for tasks such as checking out internal submodules, downloading private packages via managers like Bundler, deploying applications to servers (including Heroku), executing remote commands, or using rsync to synchronize files.

Variable Configuration Standards

The most secure and supported method for managing SSH keys is by utilizing the CI/CD variables feature. There are two primary ways to implement this:

• File-type Variables: In this configuration, the variable is stored as a file on the runner's filesystem. This is the recommended approach because SSH keys contain whitespace characters that can be corrupted or misinterpreted if stored as regular string variables.
• Regular Variables: The key is stored as a string and must be manually echoed into a file or piped into ssh-add.

When configuring these variables, specifically for a variable named SSH_PRIVATE_KEY, the following settings are mandatory for operational success:

• Visibility: Must be set to Visible. This is required because masked variables cannot contain whitespace characters, which are inherent to SSH private keys.
• Value Ending: The value must end with a newline (LF character). Users should press Enter or Return at the end of the last line before saving the variable to prevent formatting errors during the injection process.
• Masking Restrictions: Since the key cannot be masked due to whitespace, it is highly recommended to restrict these variables to protected branches or tags. This prevents unauthorized users from modifying a CI job to output the key in plain text via the job logs.

Security Best Practices

The use of SSH keys in automated environments introduces specific security risks. To mitigate these, the following protocols must be observed:

• Avoid Personal Keys: Never reuse a personal SSH key for automated CI/CD jobs. Generate a dedicated key pair specifically for the deployment process.
• Key Rotation: Regularly rotate SSH keys to minimize the impact of a potential leak.
• Log Protection: Avoid using commands like cat or tee on the SSH_PRIVATE_KEY variable, as this will expose the private key in the job logs.
• Pipeline Visibility: Be mindful of who has access to view the pipeline logs, as debug logging may accidentally reveal sensitive information.

Deploy Keys vs. Project Keys

In scenarios where the remote server needs to pull code directly from a GitLab repository, the concept of a Deploy Key becomes relevant. This is distinct from the SSH key used by the Runner to access the server.

Deploy keys can be scoped in two ways:
- Project deploy key: Access is limited to the specific project where the key is added.
- Public deploy key: Access can be granted to any project across the entire GitLab instance.

If a pipeline is designed to log into a server and then run git pull from a private repository, the public key of the server must be added as a deploy key in the GitLab project settings.

Pipeline Configuration and Implementation

The .gitlab-ci.yml file defines the logic of the pipeline. The implementation of an SSH deployment typically requires a before_script section to prepare the environment and a script section to execute the remote commands.

The Setup Phase (before_script)

The before_script is critical because it transforms the ephemeral Docker container into an SSH client. The following sequence is required:

1. Install SSH Client: In Alpine-based images, the openssh package must be installed.
2. Initialize SSH Agent: The ssh-agent must be started to manage the private key.
3. Inject Private Key: The SSH_PRIVATE_KEY variable is passed to ssh-add.
4. Directory Preparation: The .ssh directory must be created with restrictive permissions (chmod 700).
5. Trusting the Host: To avoid the "Host authenticity cannot be established" interactive prompt, ssh-keyscan is used to add the remote server's IP address to the known_hosts file.

Implementation Example 1: Basic Command Execution

This configuration focuses on verifying the connection and creating a simple file on the remote host.

```yaml
image: alpine:latest

pages:
stage: deploy
beforescript:
- 'command -v ssh-agent >/dev/null || ( apk add --update openssh )'
- eval $(ssh-agent -s)
- echo "$SSHPRIVATEKEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan $VMIPADDRESS >> ~/.ssh/knownhosts
- chmod 644 ~/.ssh/knownhosts
script:
- ssh $SSHUSER@$VMIPADDRESS "hostname && echo 'Welcome!!!' > welcome.txt"
artifacts:
paths:
- public
only:
- master
```

Implementation Example 2: Git-Based Deployment

In more complex scenarios, the pipeline triggers a git pull on the remote server to update the source code.

```yaml
stages:
- deploy

deploy:
image: alpine:latest
stage: deploy
only:
- prelive
beforescript:
- apk update
- apk add openssh-client
- install -m 600 -D /dev/null ~/.ssh/idrsa
- echo "$SSHPRIVATEKEY" | base64 -d > ~/.ssh/idrsa
- ssh-keyscan -H $SSHHOST > ~/.ssh/knownhosts
script:
- ssh $SSHUSER@$SSHHOST "cd $WORKDIR && git checkout $PRELIVEBRANCH && git pull && exit"
afterscript:
- rm -rf ~/.ssh
```

Troubleshooting Common Failures

Despite correct logic, several common failures can occur during SSH deployment.

Permission Denied (publickey, password)

This is the most frequent error. It occurs when the remote server rejects the authentication attempt. Potential causes include:

• Public Key Not Authorized: The public key associated with the SSH_PRIVATE_KEY has not been added to the ~/.ssh/authorized_keys file on the remote server.
• Incorrect Key Permissions: The .ssh directory on the server or the authorized_keys file has permissions that are too open, causing SSH to reject the key for security reasons.
• Variable Formatting: The private key was pasted into the GitLab variable without a trailing newline, or it contains carriage returns (\r) from a Windows environment, which can be mitigated using the tr -d '\r' command.

Libcrypto and Key Loading Errors

Errors such as Load key "/root/.ssh/id_rsa": error in libcrypto typically indicate that the private key is not in a format the SSH client recognizes, or it is corrupted during the injection process. This is often solved by ensuring the key is correctly base64 encoded/decoded if that method is used, or by using the ssh-agent to load the key directly from a variable.

Directory Not Found Errors

Users may encounter /usr/bin/bash: line 157: /root/.ssh/id_rsa: No such file or directory. This occurs when the pipeline attempts to write a key to a directory that does not exist. The solution is to explicitly create the directory using mkdir -p ~/.ssh before attempting to write the key file.

Technical Specifications Summary

The following table summarizes the operational requirements for a successful GitLab CI SSH deployment.

Conclusion

Implementing SSH deployments within GitLab CI/CD requires a meticulous approach to environment preparation and security. The process is essentially a bridge between a transient execution environment (the GitLab Runner) and a persistent target (the remote server). By utilizing before_script to dynamically configure the SSH client, employing ssh-keyscan to handle host verification, and utilizing secure CI/CD variables for key storage, organizations can achieve a robust automation flow. The transition from manual deployments to this automated model significantly reduces the risk of human error and ensures that the state of the remote server is always synchronized with the state of the version-controlled repository. The critical path to success lies in the strict adherence to SSH permission standards and the correct handling of the private key's formatting within the GitLab interface.

Sources

1. GitLab CI/CD Pipeline Run Script via SSH to Remote Server
2. Using SSH keys with GitLab CI/CD
3. Deploying code from GitLab repository into remote server
4. Deploy keys Documentation
5. Using GitLab CI to Deploy via SSH