HashiCorp Vault and GitLab CI/CD Integration

The synchronization of secure secret management with continuous integration and continuous deployment pipelines represents a critical juncture in modern DevOps architecture. By integrating HashiCorp Vault with GitLab CI/CD, organizations transition from static, high-risk environment variables to a dynamic, centralized secret orchestration system. This architecture ensures that sensitive data—such as API keys, database passwords, and certificates—is not stored in plain text within the version control system or the CI/CD configuration files, but is instead fetched just-in-time during the execution of a job. The integration relies on a trust relationship where the GitLab server acts as an identity provider, issuing JSON Web Tokens (JWT) that Vault can validate to grant specific access based on predefined policies. This mechanism eliminates the need for long-lived secrets and reduces the attack surface by implementing the principle of least privilege.

Vault Server Initialization and Unsealing

The foundational step in deploying HashiCorp Vault is the installation and the subsequent initialization of the server. On a Debian-based system, the process begins with the addition of the HashiCorp GPG key and the official repository to ensure the software is authentic and up-to-date.

The installation sequence involves the following commands:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com focal main"
apt-get update && apt-get install vault
systemctl start vault

Once the service is active, the operator must initialize the Vault instance. This process generates a set of unseal keys and an initial root token.

vault operator init

This command produces five unseal keys and one root token. For example, the system might output:

Unseal Key 1: b8P+huX0Vg8pEJeyJl+oeDPyhpy6QfhXsvMx6rPFHKaT
Unseal Key 2: fYAydRBmZIFO4V/QXe4YBZ6ow3L2MqK6tbB+SGBBA1Px
Unseal Key 3: QggzBeKmJJAU7vignPA9emKFppD7Sov8VWUc8g7kytr3
Unseal Key 4: SRTc/JCxVZ9M9jYwTOrAHhbM6ehHtpQ9WU8/rIT leverages 24h
Unseal Key 5: B24sVrIpnaea2FJEB4NISisNtTYUYoi1S5MFJpmL5W0W
Initial Root Token: hvs.mSX4zcy6M7suKKnnSguIg5j6

The unseal keys are a security mechanism designed to prevent a single operator from gaining full access to the Vault. Vault starts in a sealed state, meaning the encryption keys are encrypted. To unseal the Vault, a threshold of keys (typically three out of five) must be provided.

The unsealing process is executed via:

vault operator unseal <unseal_key_1>
vault operator unseal <unseal_key_2>
vault operator unseal <unseal_key_3>

After unsealing, the administrator must authenticate using the root token to begin configuring the system:

vault login <root_token>

The impact of this process is the creation of a secure, encrypted vault that requires a quorum of keys to become operational, preventing unauthorized access during boot-up or recovery. This connects directly to the overall security posture of the CI/CD pipeline, as the root of trust starts with these keys.

Secret Engine Configuration and Policy Definition

Once the Vault is unsealed and the operator is authenticated, the next phase is enabling the secret engine and defining the access policies. The Key-Value (KV) store is the most common engine used for storing static secrets like passwords and keys.

To enable the KV-v2 secret engine at a specific path:

vault secrets enable -path= secret kv-v2

Alternatively, for local testing purposes, a secret engine can be enabled at a different path:

vault secrets enable -path=local kv

Once the engine is active, secrets can be stored. For instance, creating secrets for AWX and ArgoCD applications:

vault kv put secret/gitlab/awx login = "admin" password = 'adminpassword'
vault kv put secret/gitlab/argocd login = "admin" password = 'adminpassword'

Or for different datasets:

vault write local/esdata AWS_KEY="AKIAIOSFODNN7EXAMPLE" AWS_PASS="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" SAT_ID="22" ENCR_KEY="qwerty123"

To control who can access these secrets, Vault uses policies. A policy file, such as gitlab-policy.hcl, must be created to define the capabilities granted to the GitLab integration.

sudo vim /root/gitlab-policy.hcl

The content of the policy file should be:

path "secret/data/gitlab/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}

This policy grants full CRUD (Create, Read, Update, Delete) and List capabilities to any secret stored under the secret/data/gitlab/ path. The policy is then loaded into Vault:

vault policy write gitlab-policy gitlab-policy.hcl

The impact of this configuration is the creation of a granular permission boundary. Instead of granting global access, the policy ensures that the GitLab CI/CD process can only interact with secrets specifically designated for its use. This prevents a breach in one application from exposing secrets belonging to other departments or projects.

GitLab CI/CD Integration and Authentication

The integration between GitLab and Vault allows the CI/CD pipeline to authenticate with Vault without requiring hardcoded credentials in the .gitlab-ci.yml file. This is achieved through the use of JSON Web Tokens (JWT).

GitLab provides a JWT for every job, which can be used for authentication with a Vault server configured for the JWT authentication method. The Vault server is provided with the GitLab instance's base URL (e.g., https://gitlab.example.com) as the oidc_discovery_url. This allows Vault to retrieve the necessary keys to validate the token from the GitLab instance.

The Vault server can use bound claims to match against the JWT claims, restricting which secrets each specific CI/CD job has access to based on project, branch, or environment.

For simpler integrations, an access token can be created manually for GitLab:

vault token create -policy= gitlab-policy -period= 24h

This token is then used in the GitLab CI/CD project settings as environment variables:

• VAULT_ADDR = https://<vault_server_ip>:8200
• VAULT_TOKEN = <vault_token>

By defining these in the project settings, they are automatically available to the runner, eliminating the need to declare them within the .gitlab-ci.yml file.

The real-world consequence of this setup is the removal of "secret leakage" in logs or version control. When the runner initiates a job, it uses these variables to communicate with the Vault API, ensuring that the actual secret values are only resident in the runner's volatile memory during the job's execution.

Implementing Secret Retrieval in Pipelines

Retrieving secrets within the pipeline script requires interacting with the Vault API. This is typically done using curl and processed using jq to extract specific fields from the JSON response.

To export ArgoCD credentials from Vault:

export ARGOCD_SECRET=$(curl --silent --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/data/gitlab/argocd)
export ARGOCD_USERNAME=$(echo $ARGOCD_SECRET | jq -r '.data.data.login')
export ARGOCD_PASSWORD=$(echo $ARGOCD_SECRET | jq -r '.data.data.password')

Similarly, for AWX passwords:

AWX_PASSWORD=$(echo $AWX_SECRET | jq -r '.data.data.password')

In a complex build environment, such as one involving Docker, there are specific considerations regarding when variables are available. For example, variables exported in the before_script section are not available during the docker build process because the image is built before the script runs. To solve this, credentials like NPM_USER and NPM_PASS must be defined as CI/CD variables at the project level and passed as build arguments.

Example of a build_and_test_awx job configuration:

yaml build_and_test_awx: stage: build_and_test tags: - docker1 image: docker:latest services: - name: docker:dind variables: DOCKER_DRIVER: overlay2 DOCKER_HOST: "tcp://docker:2375" DOCKER_TLS_CERTDIR: "" script: - git clone --single-branch --branch $BRANCH $REPO_URL - docker build --build-arg NPM_USER="${NPM_USER}" --build-arg NPM_PASS="${NPM_PASS}" -t $DOCKER_IMAGE -f Dockerfile . - docker run

The use of docker:dind (Docker-in-Docker) and the overlay2 driver ensures that the environment is optimized for building images within the GitLab runner. The impact of this specific implementation is that it separates the "secret retrieval" phase from the "build" phase, ensuring that secrets are passed securely as build arguments rather than being baked into the image layers in plain text.

Security Analysis of Secret Distribution

A common point of contention in CI/CD architecture is where the secret should reside during the deployment process. There is a tension between giving the GitLab server access to the secret and giving the runner access.

One approach is to avoid giving the GitLab server the token entirely and instead authenticate the server the runner is operating on. This would restrict secret access to the specific user account on the server. For example, using Podman to run a container while fetching a secret directly from Vault:

export VAULT_ADDR=https://vault-test.my-org.com
podman run -itd \
-v ${PWD}/web:/var/www/html/:z \
--name phptest \
--network helpdesk-network \
-p 8081:80 \
-e "MYVAR=$(vault kv get -field testkey kv/mysecrettest)" \
php:apache-bullseye

In this scenario, the runner deploying the container fetches the secret value, assuming a vault login was performed on that server. This prevents the GitLab server from ever possessing the value.

However, the prevailing architectural view is that the GitLab server acts as the "passport office." Because the GitLab server is the ultimate arbiter of which runner is executing which job, it must possess the private keys to create the JWTs that Vault trusts. If a user has access to the GitLab project, they could potentially trigger a CI/CD job to output a secret value regardless of whether the token is held by the server or the runner.

To mitigate this, the following security strategies are recommended:

• Restrict CI/CD pipelines that run user-modifiable code to low-privilege, read-only secrets.
• Implement a review and approval process for code before it reaches the CD stage.
• Use non-privileged accounts on the runner server.
• Implement a structure where a non-privileged runner account has sudo access to run a specific script or file as another user who possesses the secrets.

The consequence of this design is that security is not just about the location of the token, but about the governance of the pipeline. The "passport office" analogy illustrates that the authority to grant access is as critical as the access itself.

Technical Implementation Summary

The deployment of a secure secret management system involving GitLab and Vault requires a multi-layered approach. This begins with the infrastructure setup, moves through policy definition, and concludes with the integration of the runner.

The following table summarizes the key technical components:

The final stage of implementation often involves the use of Ansible for automating the runner setup. Commands for installing the GitLab runner include:

curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
dpkg -i gitlab-runner_amd64.deb
cp gitlab-runner/config.toml /etc/gitlab-runner/config.toml
systemctl restart gitlab-runner

This automation ensures that the environment is consistent across multiple runners, which is essential for maintaining the security integrity of the Vault integration.

Conclusion

The integration of HashiCorp Vault with GitLab CI/CD transforms the security paradigm of automated pipelines from one of "stored secrets" to "dynamic identity." By leveraging the JWT authentication method, organizations can ensure that secrets are never stored in plain text, but are instead provided to the runner based on a cryptographically verified identity. The process requires a rigorous setup—from the initial unsealing of the Vault to the definition of granular HCL policies—but the result is a system where the "passport office" (GitLab) can securely authorize the "traveler" (the runner) to access the "vault" (HashiCorp Vault).

The analysis shows that while there are various methods to distribute secrets—such as fetching them directly on the runner via Podman or using the GitLab server as an intermediary—the most robust approach combines least-privilege policies with strict code review processes. The potential for secret leakage via user-modifiable code is a systemic risk that cannot be solved by token location alone, but rather by restricting the privileges of the identities that trigger the pipelines. Ultimately, the transition to a Vault-based architecture reduces the risk of catastrophic credential exposure and provides a scalable, auditable framework for modern DevOps infrastructure.

Sources

1. Secure Secrets Management using HashiCorp Vault with GitLab CI/CD
2. GitLab Documentation - HashiCorp Vault
3. HashiCorp Discuss - Vault Secrets in CI/CD
4. GitLab Vault Demo GitHub