Integrating SSH Authentication within GitLab CI/CD Pipelines

The implementation of Secure Shell (SSH) authentication within a GitLab CI/CD pipeline is a critical requirement for modern DevOps workflows. In an automated environment, the GitLab Runner acts as the orchestrator, but it often lacks the inherent credentials necessary to interact with external systems. Whether the objective is to deploy code to a production server, fetch private submodules, or execute remote commands on a Cloud Platform environment, the secure transmission of SSH keys is the fundamental mechanism that enables these operations. Because GitLab does not provide a native, built-in management system for SSH keys within the runner's execution environment, developers must manually inject these credentials using CI/CD variables and configure the environment—typically a Docker container—to utilize them during the job's lifecycle.

Use Cases for SSH Keys in CI/CD

Integrating SSH capabilities into a pipeline is not merely about deployment; it serves several architectural purposes across the software development lifecycle.

• Checking out internal submodules: Many projects rely on submodules stored in private repositories. Without a valid SSH key, the GitLab Runner cannot authenticate with the version control system to clone these dependencies, causing the build to fail at the initial stage.
• Downloading private packages: Certain package managers, such as Bundler for Ruby, require SSH authentication to pull private gems or libraries that are not hosted on public registries.
• Application deployment: The most common use case involves pushing the compiled application or containerized artifact to a remote server, such as a private VPS or platforms like Heroku, using secure tunnels.
• Remote command execution: Pipelines often need to trigger scripts on a remote server, such as deploy.sh, to restart services, clear caches, or run database migrations after a successful build.
• File synchronization: Utilizing rsync allows the build environment to efficiently transfer only the changed files to a remote destination, reducing bandwidth and deployment time.

Security Implementation and Key Management

The handling of private keys within a CI/CD environment introduces significant security risks if not managed with extreme precision. The primary objective is to ensure that the private key is never exposed in the job logs or stored in plain text within the repository.

Key Generation and Rotation

It is a mandatory security practice to generate a dedicated SSH key pair specifically for the CI/CD pipeline. Personal SSH keys should never be reused for automated jobs. The use of dedicated keys allows for granular control; if a specific pipeline is compromised, the key can be revoked without affecting the developer's personal access. Furthermore, keys should be rotated regularly to mitigate the risk of long-term unauthorized access.

GitLab CI/CD Variable Types

GitLab provides two primary methods for storing SSH keys within the project settings: File type variables and Regular variables.

• File type variables: This is the preferred method. When a variable is set as a "File" type, GitLab stores the content and provides the job with a path to a temporary file containing the key. This approach is superior because it preserves multiline formatting and avoids the common errors associated with escaping newline characters.
• Regular variables: In this method, the key is stored as a string. To use it, the user must manually echo the variable into a file within the before_script section. This method is more prone to formatting errors and is generally discouraged compared to the file type approach.

Variable Visibility and Masking

When configuring the SSH_PRIVATE_KEY variable, the visibility must be set to "Visible". This is because SSH keys contain whitespace and newline characters, which are incompatible with GitLab's "Masked" variable requirements. Since the key cannot be masked, it is imperative that developers never use commands like cat or tee on the variable, as this would print the private key directly into the job logs, exposing it to anyone with access to the pipeline history.

Configuration for Docker Executors

When using the Docker executor, each job runs in an isolated container. This means the environment is ephemeral and lacks any pre-existing SSH keys or known host fingerprints.

Pre-requisites and Dependencies

Before an SSH connection can be established, the container must have the necessary tools installed. For Debian-based images, the openssh-client and git packages are required. The pipeline must check for the existence of the ssh-agent and install it if it is missing.

The Authentication Workflow

The standard process for establishing SSH connectivity in a Docker-based job involves the following sequence of operations:

1. Installation: Ensuring the openssh-client is present in the image.
2. Agent Activation: Running eval $(ssh-agent -s) to start the SSH agent in the background.
3. Key Injection: Loading the private key from the CI/CD variable into the agent using ssh-add.
4. Host Verification: Adding the remote server's public key to the known_hosts file using ssh-keyscan to prevent the "Host verification failed" error that occurs when a server is encountered for the first time.

Implementation via .gitlab-ci.yml

The .gitlab-ci.yml file defines the automation logic. The implementation differs slightly depending on whether the user is targeting a general server or a specific Cloud Platform environment.

General Server Example

The following configuration demonstrates a standard approach for a staging environment using a Docker image.

```yaml
image: docker:git
services:
- docker:dind
stages:
- staging

beforescript:
- docker login -u gitlab-ci-token -p $CIBUILDTOKEN $CIREGISTRY
- mkdir -p ~/.ssh
- echo "$DEPLOYSERVERPRIVATEKEY" | tr -d '\r' > ~/.ssh/idrsa
- chmod 600 ~/.ssh/idrsa
- eval "$(ssh-agent -s)"
- ssh-add ~/.ssh/idrsa
- ssh-keyscan -H 'your.server.hostname' >> ~/.ssh/known_hosts

staging:
stage: staging
tags:
- docker
only:
- staging
script:
- docker build --pull -t $CIREGISTRYIMAGE:staging .
- docker push $CIREGISTRYIMAGE:staging
- ssh $SERVERUSER@$SERVERHOSTNAME < deploy.sh
```

Acquia Cloud Platform Implementation

For those using the Acquia Cloud Platform, specific variables and configurations are required to connect to the environment before the build code stage begins.

• Required Variables: The user must navigate to Settings > CI/CD > Variables and add SSH_PRIVATE_KEY and SSH_PASSPHRASE.
• Host Configuration: The host follows a specific pattern, such as mysitedev.ssh.prod.acquia-sites.com.
• User Configuration: The connection uses a specific Cloud Platform user, formatted as [email protected].

The following configuration snippet illustrates the before_script required for this environment:

```yaml
include:
- project: 'acquia/standard-template'
file: '/gitlab-ci/Auto-DevOps.acquia.gitlab-ci.yml'

Build Code:
beforescript:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
- eval $(ssh-agent -s)
- echo 'echo $SSHPASSPHRASE' > ~/.ssh/config
```

Configuration for Shell Executors

The Shell executor operates differently because it runs directly on the host machine where the GitLab Runner is installed, rather than in an isolated container. This significantly simplifies the setup.

Local Key Generation

Instead of injecting keys via variables, the most efficient method for Shell executors is to generate the SSH key directly on the runner machine.

• Access: The user must sign in to the server running the GitLab Runner.
• User Context: The keys must be generated as the gitlab-runner user using the command sudo su - gitlab-runner.
• Pair Generation: A new SSH key pair is generated. It is crucial not to add a passphrase to this key; otherwise, the before_script will hang while waiting for manual input, causing the job to time out.

Host Fingerprint Acceptance

After generating the key, the user must manually connect to the remote server once from the terminal to accept the server's fingerprint:

ssh example.com

This ensures that subsequent automated attempts to connect via the pipeline do not fail due to the interactive prompt asking to verify the authenticity of the host.

Technical Comparison of Executor Methods

The choice between Docker and Shell executors fundamentally changes how SSH keys are managed and deployed.

Advanced Troubleshooting and Best Practices

Ensuring a stable SSH connection in a CI/CD pipeline requires attention to detail regarding file permissions and network security.

File Permissions

SSH is designed to be secure and will ignore private keys if the permissions are too open. The private key file must be set to read/write only for the owner. This is achieved using the command:

chmod 600 ~/.ssh/id_rsa

If this command is omitted, the SSH client will throw a "permissions are too open" error and refuse to use the key.

Man-in-the-Middle (MITM) Protection

Using ssh-keyscan is a common way to bypass the manual prompt for host verification, but it can leave the pipeline vulnerable to MITM attacks if the network is compromised. To verify the private server's own public key and ensure its authenticity, it is recommended to manually verify the fingerprint of the server before adding it to the known_hosts file.

Dealing with Passphrases

If a private key is encrypted with a passphrase, the ssh-agent must be used to provide the passphrase. In the Acquia example, a script is created to echo the SSH_PASSPHRASE variable to facilitate this. However, for most standard GitLab CI/CD pipelines, it is recommended to use keys without passphrases to avoid the complexity of automated passphrase entry.

Conclusion

The integration of SSH within GitLab CI/CD pipelines is a balance between operational necessity and security rigor. By utilizing file-type CI/CD variables, managing the ssh-agent within the before_script section, and adhering to strict file permission standards, organizations can achieve a seamless and secure deployment pipeline. The distinction between Docker and Shell executors dictates the strategy—either dynamic injection for portability or static configuration for simplicity. Ultimately, the use of dedicated, rotated keys and the avoidance of masking-incompatible characters ensures that the automation remains robust without sacrificing the security of the infrastructure.

Sources

1. Acquia Docs - Using SSH during job pipeline
2. GitLab Docs - SSH keys with GitLab CI/CD
3. GitHub Gist - yannhowe/5ab1501156bd84c8ac261e2c17b8e3e0