Mastering Google Cloud Platform Authentication in Terraform

Establishing a secure and robust authentication mechanism between Terraform and Google Cloud Platform (GCP) is a foundational requirement for any Infrastructure as Code (IaC) deployment. Depending on whether the execution environment is a local developer workstation, a CI/CD pipeline, or a managed platform like HCP Terraform, the strategy for credential management varies significantly. Poorly managed credentials—such as hardcoding JSON keys in source code—introduce critical security vulnerabilities, while overly restrictive permissions can lead to deployment failures.

Effective GCP authentication requires a deep understanding of the hierarchy of credentials, ranging from Application Default Credentials (ADC) for rapid development to dynamic OIDC-based workload identities for enterprise-grade automation.

Local Development and Application Default Credentials

For engineers working on their local machines, the most efficient and secure method of authentication is the use of Application Default Credentials (ADC). This approach removes the need to manually manage JSON key files on the local filesystem, leveraging the existing authentication state of the Google Cloud SDK.

The gcloud CLI Workflow

The primary tool for initializing local authentication is the gcloud command-line interface. By using the SDK, developers can authenticate their local environment to the GCP project without exporting long-lived service account keys.

To set up User Application Default Credentials, the following command is executed:

bash gcloud auth application-default login

Once this command is executed, the Google Cloud SDK opens a browser window for the user to authenticate. Upon successful login, a credential file is stored locally in a well-known location that the Terraform GCP provider is programmed to check automatically. This allows Terraform to interact with GCP resources seamlessly without any explicit credentials attribute defined in the provider block.

Local Service Account Usage

While ADC is recommended for users, there are scenarios where a local environment must mimic a service account's specific permissions. This is achieved by creating a Service Account in the GCP Console, granting it the necessary IAM roles, and downloading the associated JSON key file.

To use this key locally, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be set to the absolute path of the downloaded JSON file:

bash export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/keyfile.json"

The Terraform GCP provider detects this environment variable and uses the referenced service account for all subsequent API calls.

Managing Credentials in HCP Terraform

HCP Terraform introduces specific constraints because it executes Terraform runs on remote agents rather than a local machine with access to your home directory or local environment variables. Referencing a local file path (e.g., credentials = file("/path/to/key.json")) is not viable in HCP Terraform as the remote agent does not have access to the local disk of the user who pushed the code.

Static Credential Method 1: Terraform Variables

One method to provide GCP credentials to HCP Terraform is by treating the entire content of the JSON key file as a Terraform variable. This approach shifts the credential from a file to a string stored within the HCP Terraform workspace.

1. Configuration: Define a variable in the Terraform code to hold the JSON string.
2. Provider Implementation: Pass this variable into the credentials argument of the google provider.

```hcl
provider "google" {
project = ""
region = ""
zone = ""
credentials = var.gcp-creds
}

variable "gcp-creds" {
description = "GCP service account credentials JSON."
type = string
default = ""
}
```

3. HCP Terraform Setup: In the workspace UI, create a variable named gcp-creds and paste the entire content of the JSON key file as the value. It is mandatory to mark this variable as sensitive in the UI to ensure the credentials are encrypted and masked in logs.

Static Credential Method 2: Environment Variables

Alternatively, HCP Terraform can be configured to use standard GCP environment variables. The GOOGLE_CREDENTIALS variable is recognized by the provider. However, because JSON files often contain newlines and spaces that can cause parsing issues in environment variable fields, the JSON must be flattened into a single line.

To format the JSON key using jq on a local machine:

bash cat file.json | jq -c

The resulting single-line string is then copied and pasted into an HCP Terraform environment variable with the key GOOGLE_CREDENTIALS. Like Terraform variables, this must be marked as sensitive.

Advanced Dynamic Credentials with OIDC

The modern enterprise standard for GCP authentication in HCP Terraform is the use of Dynamic Credentials via OpenID Connect (OIDC). This method eliminates the need for long-lived JSON keys entirely, removing the risk of key leakage and the administrative burden of key rotation.

Architectural Overview of Dynamic Credentials

Dynamic credentials establish a trust relationship between HCP Terraform and GCP. Instead of a static key, HCP Terraform requests a short-lived access token from GCP's Security Token Service (STS). This token is valid only for the duration of the terraform plan or terraform apply operation.

The setup requires a two-sided configuration:

1. GCP Configuration: A Workload Identity Pool and Provider must be created. This configuration tells GCP to trust tokens issued by HCP Terraform.
2. HCP Terraform Configuration: Specific environment variables must be set in the workspace to tell the GCP provider how to request the dynamic token.

Mandatory Configuration Variables

To enable dynamic credentials, several environment variables must be configured. These can be set at the workspace level or globally using the TFC_DEFAULT_ prefix.

If the unified TFC_GCP_WORKLOAD_PROVIDER_NAME is not used, the configuration must provide the project number, pool ID, and provider ID as separate variables.

Implementing Dynamic Credentials in Code

When using dynamic credentials, the provider block must remain clean of static credential references. However, to support complex environments—such as multiple GCP projects or aliases—HCP Terraform can supply configuration details through a specialized variable object.

```hcl
variable "tfcgcpdynamic_credentials" {
description = "Object containing GCP dynamic credentials configuration"
type = object({
default = object({
credentials = string
})
aliases = map(object({
credentials = string
}))
})
}

provider "google" {
credentials = var.tfcgcpdynamic_credentials.default.credentials
}

provider "google" {
alias = "ALIAS1"
credentials = var.tfcgcpdynamic_credentials.aliases["ALIAS1"].credentials
}
```

Critical Constraints and Warnings

Dynamic credentials are powerful but have specific limitations:
- Certificates: This integration will not function if a Terraform Enterprise instance utilizes a custom or self-signed certificate due to restrictions enforced by GCP.
- Variable Conflicts: To prevent authentication collisions, users must ensure that GOOGLE_CREDENTIALS or GOOGLE_APPLICATION_CREDENTIALS are not set when dynamic credentials are being used. If both are present, the static credentials will conflict with the OIDC flow.
- Version Requirements: If self-managing agents, certain versions are required for full support: TFC_GCP_PRINCIPAL_TYPE requires v1.28.3 or later, and TFC_GCP_WORKLOAD_PROVIDER_NAME requires v1.7.0 or later.

Comparison of Authentication Methods

Choosing the right authentication method depends on the environment and the security posture of the organization.

Troubleshooting and Configuration Logic

When configuring the google provider, it is essential to remember that while credentials handle who is accessing the API, the provider still needs to know where the resources reside. Regardless of the authentication method—be it a JSON key or a dynamic OIDC token—the provider block must always include the project and region arguments.

Common Failure Points

1. JSON Formatting: When using the GOOGLE_CREDENTIALS environment variable in HCP Terraform, failing to remove newlines via jq -c often leads to "invalid character" errors during the provider initialization phase.
2. Permissions Mismatch: A common mistake is authenticating successfully but encountering 403 Forbidden errors. This occurs when the Service Account (whether static or dynamic) has not been granted the necessary IAM roles (e.g., Compute Admin, Storage Admin) within the target GCP project.
3. Identity Pool Misconfiguration: For dynamic credentials, the most frequent error is an incorrect canonical name for the Workload Identity Provider. The name must follow the strict GCP format to be recognized by the STS.

Conclusion

The evolution of Terraform's integration with Google Cloud Platform reflects a broader industry shift toward "keyless" authentication. For individual developers, Application Default Credentials (ADC) provide the path of least resistance, allowing for rapid iteration without compromising the security of service account keys. For automation and orchestration, the transition from static JSON keys to HCP Terraform's dynamic OIDC credentials represents a significant security upgrade. By eliminating long-lived secrets, organizations can drastically reduce their attack surface and simplify the auditing of infrastructure changes.

The choice between static variables, environment variables, and dynamic credentials in HCP Terraform should be driven by the scale of the operation and the available security tooling. While static keys are easier to implement initially, the operational overhead of rotating those keys makes them unsuitable for large-scale production environments. Implementing a Workload Identity Pool, while requiring a more complex initial configuration, provides a scalable and secure foundation for managing cloud resources at scale.

Sources

1. How to set up Google Cloud GCP credentials in HCP Terraform
2. Use dynamic credentials with the GCP provider
3. Managing Google Cloud Credentials with Terraform
4. terraform-gcp-credentials