Terraform GCP Secret Manager: Runtime Secrets, State Hygiene, and Least-Privilege Access

Managing secrets in Terraform on Google Cloud is a balancing act between automation and exposure. Secrets need to exist in Secret Manager to be usable by workloads, but Terraform state is a liability if it contains plaintext values. The practice that has emerged at scale is to let Terraform own secret metadata, IAM, and lifecycle while keeping the actual secret values out of state and out of CI/CD configuration. In production workflows, that separation is coupled with Workload Identity Federation and prefix-based IAM conditions so that a repository can only reach the secrets it is allowed to use for a given locale and environment.

Secret Manager is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. It provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud. Secret Manager provides a scalable and robust solution for secrets management, ensuring encryption, access controls, and audit trails.

In simple terms the value proposition is:
- You can easily store and retrieve your secrets
- You can rotate and version your secrets
- It integrates well with other GCP services
- It provides audit logs to track access to secrets
- It encrypts your secret data before storing it
- You can use customer-managed encryption
- You can control who can access these secrets

Why use Secret Manager instead of hardcoding secrets into code or configuration files is that you can store secrets centrally, rather than hardcoding them into your code or configuration files. The approach improves security by reducing the risk of accidental exposure or unauthorized access to sensitive information. It is designed to help developers securely store, access, and distribute secrets needed by their applications and services. Think of it like a secure vault where you put all your important keys and passwords, and only the people you trust can access them. Plus, you get a complete record of who accessed what and when.

Architecture Shift from GitHub Actions Secrets to GCP Secret Manager

The first post in a short series about replacing GitHub Actions secrets with GCP Secret Manager in Terraform workflows covered the architecture change: moving Terraform-managed GitHub Actions secrets into GCP Secret Manager and using GCP IAM Conditions to enforce locale-based access. That change removed a major CI/CD bottleneck.

Across the rollout, 1,447 GitHub Actions environment secrets were removed from Terraform management.

The runtime impact was significant.

At peak, some runs had reached almost 2.5 hours.

The measurable result was clear. The team removed 1,447 GitHub Actions environment secrets from Terraform management and shifted secret retrieval to runtime through GCP Secret Manager. That reduced the GitHub API-heavy refresh pattern that had pushed workflows from sub-5-minute runs into routine 45-minute to 2.5-hour runs.

After the migration:
- NP workflow average dropped from 72m 30s to 3m 17s
- PRD workflow average dropped from 85m 12s to 9m 45s

That is roughly ~95% runtime improvement for NP and ~89% runtime improvement for PRD.

The security posture also improved:
- secrets moved into a dedicated secrets platform
- access became policy-driven through GCP IAM
- blast radius was reduced by locale-based access controls
- secret sprawl across CI/CD environments was reduced
- runtime secret exposure became more intentional

Reusable Workflow Pattern for Secret Injection

The new workflow flow replaces static GitHub secrets with runtime lookups from Secret Manager.

That gave us a cleaner platform model:
- fewer GitHub secrets
- less Terraform state bloat
- less GitHub API pressure
- smaller caller workflows
- centralized secret management
- least-privilege access control
- cleaner Terraform inputs

Business impact is clear. We removed 1,447 GitHub Actions environment secrets from Terraform management and shifted secret retrieval to runtime through GCP Secret Manager. That reduced the GitHub API-heavy refresh pattern that had pushed workflows from sub-5-minute runs into routine 45-minute to 2.5-hour runs. This started as a workflow performance problem, but the real fix was platform design.

Terraform for Secret Manager Lifecycle

Secrets management is one of the trickiest parts of infrastructure as code. You need to store database passwords, API keys, and certificates somewhere, but you definitely should not put them in your Terraform code or persist them in your Terraform state file in plain text. Google Cloud Secret Manager gives you a centralized, encrypted, access-controlled store for sensitive values, and Terraform can manage the entire lifecycle.

This guide covers creating and managing secrets with Terraform, controlling access, handling rotation, and integrating secrets with your applications.

Secret Metadata vs Secret Value

There is an inherent tension with secrets in Terraform. If you use the secretdata argument or a normal managed password resource, Terraform needs to know the secret value to store it in Secret Manager, and that value ends up in the Terraform state file. With Terraform 1.11 or later and a current Google provider, use write-only arguments such as secretdata_wo where possible. The state file itself still needs to be treated as sensitive, which means encrypted storage and restricted access.

That said, managing secret metadata with Terraform is straightforward and valuable. Defining the Secret Manager secret resource, IAM policies, replication settings with Terraform is valuable.

A typical tutorial layout using Terraform is:
- Create a service account for Artifact Registry access
- Assign needed permissions
- Create the service account key
- Create a secret with GCP Secret manager
- Create a secret version with service account key
- Set up proper IAM permissions for secret access
- Create additional secrets for our database credentials
- Test everything works properly

Setting up Secret Manager with Terraform starts with enabling required APIs first. Before we start, we need to make sure the required APIs are enabled.

Example metadata definition:

```hcl
resource "googlesecretmanagersecret" "svckey" {
secretid = "artifact-registry-sa-key"
replication {
usermanaged {
replicas {
location = "us-central1"
}
}
}
labels = {
environment = var.environment
service = "artifact-registry"
}
}

resource "googlesecretmanagersecretversion" "svckeyversion" {
secret = googlesecretmanagersecret.svckey.id
secret_data = filebase64("${path.module}/keys/sa.json")
}
```

The service account key and pull the secret with the CI workflow using gcloud commands is a common pattern. Setting up a Cloud Function with Cloud Scheduler to regularly rotate the service account key was recommended as a perfect solution.

Access Control and Safe Injection

Use resource-level IAM for secrets. Grant secretAccessor on individual secrets, not at the project level. Use the sensitive flag on Terraform variables that hold secret values to prevent them from appearing in plan output. Prefer injecting secrets as environment variables in Cloud Run and Cloud Functions over fetching them in application code. It is simpler and the platform handles injection.

Additional practices:
- Set up rotation for long-lived secrets like database passwords and API keys
- Use labels to organize and track secrets across services and environments
- Never log secret values. Be careful with debug logging that might accidentally print environment variables

Use write-only arguments where possible; if you use regular secret-value arguments, the actual sensitive data ends up in Terraform state, so ensure your state is encrypted and access-controlled.

Operational Recommendations

Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud. Secret Manager provides a scalable and robust solution for secrets management, ensuring encryption, access controls, and audit trails.

Practical controls:
- Enable audit logs to track access to secrets
- Encrypt secret data before storing it, with customer-managed encryption where required
- Control who can access these secrets via IAM conditions
- Rotate and version secrets regularly
- Integrate with other GCP services via Secret Manager integration

When deploying secrets with Terraform, the workflow authenticates to GCP using Workload Identity Federation. GCP IAM limits which Secret Manager prefixes the repo can access. The workflow builds a standard secret map for that repo type and environment. The workflow scans the Terraform directory for declared variables. Secrets without matching variables are skipped. Matching secrets are fetched from GCP Secret Manager. Terraform receives only the needed TFVAR* values.

This approach improves security by reducing the risk of accidental exposure or unauthorized access to sensitive information.

Conclusion

Google Secret Manager with Terraform gives you a centralized, auditable, and access-controlled system for managing sensitive configuration. By defining secrets, their access policies, and their integration with Cloud Run and Cloud Functions in Terraform, you ensure that security configuration is reviewed and version-controlled alongside the rest of your infrastructure. Start by migrating hardcoded secrets out of your code and environment files into Secret Manager, and expand from there.

The performance gains from moving secret retrieval to runtime are substantial. The removal of 1,447 GitHub Actions environment secrets, with non-production removal of 1,121 secrets and production removal of 326 secrets, cut Deploy NP GitHub Settings from 72m 30s to 3m 17s and Deploy PRD GitHub Settings from 85m 12s to 9m 45s. The security gains are equally important: secrets moved into a dedicated secrets platform, access became policy-driven through GCP IAM, blast radius was reduced by locale-based access controls, secret sprawl across CI/CD environments was reduced, and runtime secret exposure became more intentional. The combination of Terraform-managed metadata, write-only secret arguments, resource-level IAM, and workload identity federation creates a sustainable pattern for secrets at scale.

Sources

1. Using GCP Secret Manager Terraform Workflows Passing Only
2. LivingDevOps Terraform on Google Cloud Secret Manager
3. OneUptime How to Manage GCP Secrets in Terraform Using Google Secret Manager