The integration of HashiCorp Vault within GitLab CI/CD pipelines represents a fundamental shift from static secret management to dynamic, identity-based security. In traditional CI/CD environments, secrets are often stored as masked variables within the GitLab UI, which, while encrypted at rest, remain static and are often shared across multiple jobs without granular control. By leveraging HashiCorp Vault, an organization transitions to a centralized secrets management system that provides lease-based access, detailed audit logs, and a robust identity-based authentication mechanism. This architecture ensures that sensitive data, such as AWS keys, database passwords, and encryption tokens, never reside permanently within the GitLab environment but are fetched just-in-time during the execution of a specific job. The synergy between GitLab's JSON Web Token (JWT) and Vault's authentication backend allows for a seamless, secure handshake that eliminates the need for long-lived root tokens in pipeline configurations.
HashiCorp Vault Server Deployment and Installation
The foundation of a secure secrets pipeline begins with the deployment of a dedicated Vault server. To maintain a high security posture, Vault should be installed on a separate, hardened server rather than sharing resources with the GitLab Runner or the application server. This isolation ensures that a compromise of the execution environment does not lead to an immediate compromise of the secrets engine.
For Debian-based distributions, including Ubuntu, the installation process requires the setup of the official HashiCorp repository to ensure the retrieval of the latest stable binaries and security patches. The process is executed through the following sequence of commands:
bash
sudo apt -y install gnupg2
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $( lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt -y install vault
For users operating within the CentOS or RHEL ecosystem, the installation utilizes the yum-utils package to manage the repository configuration before installing the Vault binary:
bash
sudo yum install -y yum-utils gnupg2
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install vault
Once the installation is complete, it is imperative to verify that the binary is correctly installed and accessible in the system path by checking the version:
bash
vault --version
SSL Certificate Configuration and SAN Validation
A critical failure point in Vault installations is the "tls: failed to verify certificate" error, specifically when the error message indicates that the certificate does not contain any IP Subject Alternative Names (SANs). This occurs when the SSL certificate used by Vault is generated without specifying the IP address of the server in the SAN field, causing the client (the GitLab Runner) to reject the connection for security reasons.
To resolve this, administrators must generate a new SSL certificate that explicitly includes the IP address of the Vault server in the SAN field. This ensures that the identity of the server is cryptographically verified during the TLS handshake.
Establishing Trust Between GitLab Runners and Vault
For a GitLab Runner to communicate securely with a Vault server, it must trust the certificate authority (CA) that issued the Vault server's certificate. Without this trust relationship, the Runner will terminate the connection attempt to prevent potential man-in-the-middle attacks.
The process of establishing this trust involves downloading the certificate from the Vault server and importing it into the Runner's trusted store.
The certificate can be extracted directly from the server using the following OpenSSL command:
bash
echo -n | openssl s_client -connect 10.10.0.150:8200 | openssl x509 > vault.crt
Once the vault.crt file is obtained, it must be moved to the system's trusted certificates directory and the CA store must be updated:
bash
sudo cp vault.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
To validate that the connection is now secure and the certificate is recognized, administrators should perform two tests. First, using OpenSSL to verify the handshake:
bash
openssl s_client -connect 10.10.0.150:8200 -CAfile /usr/local/share/ca-certificates/vault.crt
Second, using curl to check the health of the Vault API, which confirms that the network path and SSL trust are fully operational:
bash
curl --cacert /usr/local/share/ca-certificates/vault.crt https://10.10.0.150:8200/v1/sys/health
GitLab CI/CD Pipeline Variable Configuration
While the goal is to move secrets into Vault, certain orchestration variables must still be defined within the GitLab CI/CD pipeline settings to tell the Runner where the Vault server is located and how to authenticate. These variables are passed automatically to all jobs in the pipeline.
The following table outlines the essential variables required for the integration:
| Variable Name | Purpose | Example Value |
|---|---|---|
| VAULT_ADDR | The network address and port of the Vault server | https://192.168.1.99:8200 |
| VAULT_TOKEN | The authentication token for Vault (if not using JWT) | hvs.mSX4zcy6M7suKKnnSguIg5j6 |
It is a security best practice to keep the VAULT_TOKEN out of the Git repository and instead define it as a masked and protected variable within the GitLab UI.
Integrating Vault Secrets via .gitlab-ci.yml
GitLab provides a native secrets keyword within the .gitlab-ci.yml configuration. This allows the Runner to automatically authenticate with Vault using an ID token and fetch specific secrets before the script execution begins.
ID Token Implementation
When a job defines an id_tokens block, the Runner generates a JSON Web Token (JWT) and uses it to authenticate with Vault. This is significantly more secure than using a static token.
Example configuration for fetching a database password:
yaml
job_using_vault:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
secrets:
DATABASE_PASSWORD:
vault: production/db/password@ops
token: $VAULT_ID_TOKEN
In this configuration, the string production/db/password@ops is parsed as follows:
- production/db: This is the path to the secret.
- password: This is the specific field within the secret.
- ops: This is the path where the secrets engine is mounted.
- The resulting internal path becomes ops/data/production/db.
Secret Handling: Files vs. Variables
By default, GitLab saves the fetched secret into a temporary file and stores the path to that file in the environment variable (e.g., DATABASE_PASSWORD). This prevents the secret from being accidentally printed to the console logs.
If the secret is required as a direct string value rather than a file path, the file: false option must be explicitly set:
yaml
secrets:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
DATABASE_PASSWORD:
vault: production/db/password@ops
file: false
token: $VAULT_ID_TOKEN
Secrets Engine Support and Versioning
GitLab Runner supports various secrets engines, with the Key-Value (KV) engine being the most common.
| Secrets Engine | secrets:engine:name Value | Runner Version | Details |
|---|---|---|---|
| KV secrets engine - version 2 | kv-v2 |
13.4 | This is the default engine used by the Runner |
Custom Containerized Secret Retrieval
In advanced scenarios, a custom Docker container can be developed to handle complex secret retrieval and parsing. This approach is useful when multiple secrets must be fetched and processed before being passed to the application.
A specialized container can be built to call the Vault API and parse values such as AWS_KEY, AWS_PASS, SAT_ID, and ENCR_KEY from a specific path, such as local/esdata. This container is designed to accept arguments from the shell to return either all requested values at once or specific values one by one.
The typical pipeline flow for this custom container approach is as follows:
- Build docker image: The container is compiled and prepared.
- Push docker image: The image is uploaded to the GitLab Docker Registry.
- Get secrets from Vault: The custom container is executed to retrieve credentials.
- Passing credentials: The retrieved values are passed to subsequent stages.
Managing Secret Persistence Across Pipeline Stages
Environment variables created during a GitLab CI/CD job are volatile; they are lost immediately after the job finishes. To transfer secrets retrieved from Vault in one stage to a subsequent stage, the GitLab artifacts system must be used.
The recommended workflow is:
1. Fetch the secret from Vault during a specialized "Get Secrets" stage.
2. Save the secret value into a file.
3. Define that file as an artifact in the .gitlab-ci.yml file.
4. The subsequent stage then collects the artifact and reads the file to obtain the credential.
Security Restrictions and Role-Based Access Control (RBAC)
When configuring Vault roles for JWT authentication, it is critical to implement strict restrictions. Without these constraints, any JWT generated by the GitLab instance could potentially authenticate as any role in Vault, leading to a massive security vulnerability.
Administrators can specify several attributes for the resulting Vault tokens to limit the blast radius:
- Time-to-live (TTL): Ensures the token expires quickly.
- IP address range: Restricts the token's usability to specific network segments.
- Number of uses: Limits the token to a single use (single-use tokens).
Conclusion
The integration of HashiCorp Vault with GitLab CI/CD transforms the pipeline from a simple automation sequence into a secure, identity-driven delivery system. By removing the reliance on static variables and implementing JWT-based authentication, organizations can achieve a "Zero Trust" architecture where secrets are only available to the specific job that requires them and only for the duration of that job's execution. The requirement for proper SSL/TLS configuration, specifically the inclusion of Subject Alternative Names (SANs) in certificates, emphasizes the necessity of a rigorous security foundation. Furthermore, the ability to toggle between file-based and variable-based secret delivery and the use of artifacts for cross-stage secret passing provides the flexibility needed for complex microservices deployments. Ultimately, the transition to this model reduces the risk of credential leakage and provides a comprehensive audit trail of who accessed which secret and when, fulfilling the most stringent compliance requirements for modern software infrastructure.