Managing Google Cloud API Enablement with Terraform: google_project_service and Internal API Boundaries

In the landscape of infrastructure as code, the interaction between Terraform and cloud providers is often reduced to a simple exchange of create and delete requests. However, the Google Cloud Platform ecosystem introduces a layer of complexity that distinguishes it from other major clouds: the strict separation between public and private APIs, and the specific mechanics of service enablement. The google_project_service resource is the primary mechanism for managing the lifecycle of these public APIs within a Terraform workflow. Understanding this resource requires more than memorizing HCL syntax; it demands a comprehensive grasp of how Google Cloud structures its service surface, the distinction between enabling a service for customer use and internal service operations, and the correct application of Terraform import mechanisms for resources that already exist. This analysis explores the technical underpinnings of google_project_service, clarifies the critical boundaries between public and private APIs, and provides expert guidance on implementing API management at scale using the Project Factory module.

The Technical Role of googleprojectservice

The google_project_service resource serves a specific and singular purpose within the Terraform ecosystem for Google Cloud: it verifies and manages the enablement state of a specified public API for a given Google Cloud project. In Google Cloud architecture, creating a resource such as a Compute Engine instance is not merely a matter of defining the instance properties in code. It is a two-step process. First, the underlying service, represented by an API endpoint, must be activated for the project. Second, the specific resource can be created within the context of that enabled service.

The google_project_service resource addresses the first step. When a Terraform configuration includes this resource, it interacts with the Google Service Usage API to ensure that the target API is enabled. If the API is already enabled, the resource is a no-op. If the API is disabled, the resource initiates the enablement process. This action grants the project the necessary permissions and access rights to utilize the functionalities of that specific service. For instance, before a Terraform configuration can successfully create a google_compute_instance resource, the compute.googleapis.com API must be enabled. Without this prerequisite, any attempt to create a Compute Engine instance will result in a permission error or an "API not enabled" failure, regardless of how correctly the instance properties are defined.

The resource is defined with specific attributes that control its behavior. The project attribute specifies the Google Cloud project ID where the API should be enabled. The service attribute defines the specific API to enable, such as compute.googleapis.com. A critical attribute for disaster recovery and operational stability is disable_on_destroy. This boolean flag determines whether the API should be disabled when the Terraform resource is destroyed. Setting this attribute to false is a common best practice in production environments. Disabling a service upon the deletion of a Terraform state entry can have cascading negative effects, potentially breaking other services that depend on the enabled API or causing unnecessary latency if the service is re-enabled later. By setting disable_on_destroy = false, operators ensure that the enablement state persists beyond the lifecycle of the specific Terraform resource definition, adhering to the principle of infrastructure persistence.

hcl resource "google_project_service" "compute_api" { project = "your-gcp-project-id" service = "compute.googleapis.com" disable_on_destroy = false }

This resource is exclusively designed for managing the enablement state of publicly accessible Google Cloud APIs. It is not a generic service activation tool that works for all endpoints associated with a Google Cloud product. The google_project_service resource is strictly bound to the public API surface. Attempting to use this resource to enable internal or private APIs will result in errors. This limitation is not a bug or a restriction of the Terraform provider; it is a fundamental architectural boundary in Google Cloud. Private APIs are not exposed through the public API surface for management purposes, and therefore, they cannot be targeted by the google_project_service resource.

Public versus Private Google Cloud APIs

To use google_project_service effectively, one must understand the dichotomy between public and private APIs in Google Cloud. Google Cloud services expose a wide variety of APIs that allow applications and tools to interact with and manage resources. These APIs fall into two distinct categories based on their purpose, documentation, and exposure.

Public APIs are the primary interfaces for customers and third-party tools. They are the endpoints that the google Terraform provider is built to interact with. These APIs are well-documented, have defined REST endpoints, and are intended for external consumption. They allow customers to create, configure, and manage resources such as Compute Engine instances, Cloud Storage buckets, and BigQuery datasets. Examples of public APIs include compute.googleapis.com, storage.googleapis.com, and bigquery.googleapis.com. These are the APIs that should be enabled using google_project_service.

Private, or internal, APIs serve a completely different function. These APIs are internal to Google Cloud services and are used by Google itself for the internal operation, orchestration, and provisioning of its managed services. They expose functionalities that are not meant for direct customer interaction or management. Private APIs are generally not publicly documented, do not have stable external endpoints, and are not designed for direct access by third-party tools like Terraform. They are implementation details of the service.

A specific example of a private API is dataproc-control.googleapis.com. This API is used by the Managed Service for Apache Spark for its operational control plane. Customers do not directly interact with or manage this API. If a customer encounters a reference to dataproc-control.googleapis.com in logs, documentation, or error messages, it is crucial to recognize that this is an internal component. No customer action is required to enable this API. It is not necessary to use google_project_service to enable dataproc-control.googleapis.com, nor should one attempt to do so. Doing so will result in errors because the API is not intended for public management. These internal APIs are crucial for the internal operation of Google Cloud services, but they remain invisible to the customer’s infrastructure management tools unless the service itself is enabled through its public counterpart.

The confusion between these two types often arises when customers see internal API names in error logs or system messages and assume they need to enable them manually. The key takeaway is that if an API is identified as a private or internal Google Cloud API, no customer action is required. These APIs are managed internally by Google, and their enablement is a side effect of enabling the public service that utilizes them.

API Enablement versus Resource Import

A fundamental concept in using Terraform effectively with Google Cloud is the distinction between "enabling an API" and "importing a resource." These are two distinct operations that serve different purposes and are often conflated by new users, leading to misunderstandings about Terraform's capabilities and limitations.

Enabling an API refers to activating a specific Google Cloud service for a project. This action grants the project the necessary permissions and access to use the functionalities of that service and create resources managed by it. In the Terraform context, this is typically done using the google_project_service resource. It is a prerequisite step. You cannot create a resource if the service that supports it is not enabled.

Importing a resource, on the other hand, refers to bringing an existing cloud resource under Terraform's management. This occurs when a resource was created manually, via the Google Cloud Console, or by another process outside of Terraform. When you import a resource, Terraform generates a state entry for it, allowing you to manage its lifecycle, including updates and deletion, using your Terraform configuration. This is achieved using the terraform import command, or by utilizing import blocks introduced in Terraform version 1.5 and later.

The purpose of importing is to gain control over resources that were not initially provisioned by Terraform. For example, if a Virtual Machine was created manually in the Google Cloud Console, a Terraform configuration can be written to describe that VM. However, Terraform does not know the relationship between the configuration and the existing resource until the resource is imported.

bash terraform import google_compute_instance.my_instance projects/your-gcp-project-id/zones/us-central1-a/instances/my-vm

In this example, the terraform import command associates the existing VM named my-vm with the google_compute_instance resource defined in the Terraform code. Once imported, any subsequent changes to the Terraform configuration will be applied to the existing VM. Conversely, deleting the resource from the Terraform configuration will result in the VM being deleted from the cloud.

It is critical to note that importing does not enable APIs. Importing a resource assumes that the API for that service is already enabled. If the API is not enabled, the import may succeed if the resource exists, but any subsequent attempts to modify the resource via Terraform may fail if the API status changes or if permissions are revoked. Furthermore, you cannot import a private API. Since private APIs are not resources that can be managed by Terraform, the concept of importing them is invalid. Importing applies to resources, not to service enablement states or internal API endpoints.

Scaling API Management with the Project Factory Module

While the google_project_service resource is effective for individual projects, managing API enablement at scale across multiple projects presents significant operational challenges. Manually defining google_project_service resources for every possible API in every project is inefficient and prone to error. This is where the Terraform Project Factory module, specifically the terraform-google-project-factory module, becomes a critical tool.

This module allows users to create opinionated Google Cloud Platform projects. It creates projects and configures aspects such as Shared VPC connectivity, IAM access, Service Accounts, and API enablement to follow best practices. The module is designed to standardize project creation, ensuring that all projects have the necessary APIs enabled according to a defined policy. It is intended for use with Terraform version 1.3 and later, and has been tested using Terraform 1.10 and newer.

To operate the Project Factory, specific prerequisites must be met. The module uses the Google Terraform provider to authenticate all Google Cloud API calls. Therefore, both the google and google-beta providers must be configured with valid credentials.

```hcl
provider "google" {
credentials = "${file(var.credentials_path)}"
}

provider "google-beta" {
credentials = "${file(var.credentials_path)}"
}
```

The execution of the Project Factory requires a Service Account with specific roles. These roles grant the necessary permissions to create projects, manage IAM policies, and enable APIs. The required roles include roles/resourcemanager.folderViewer on the target folder, roles/resourcemanager.organizationViewer on the organization, roles/resourcemanager.projectCreator on the organization, and roles/billing.user on the organization. Additionally, roles/storage.admin is required on the bucket project for state storage. If Shared VPC is being used, additional roles such as roles/compute.xpnAdmin and roles/compute.networkAdmin on the organization are necessary.

A helper script is included with the module to streamline the setup process. This script creates the Seed Service Account in the Seed Project, grants the necessary roles to the Seed Service Account, and enables the necessary APIs in the Seed Project. The script is executed as follows:

bash ./helpers/setup-sa.sh -o <organization id> -p <project id> [-b <billing account id>] [-f <folder id>] [-n <service account name>]

To execute this script, the account used must have specific permissions, including resourcemanager.organizations.list, resourcemanager.projects.list, billing.accounts.list, iam.serviceAccounts.create, iam.serviceAccountKeys.create, resourcemanager.organizations.setIamPolicy, resourcemanager.projects.setIamPolicy, and serviceusage.services.enable on the project service management services. The services that must be enabled include cloudresourcemanager.googleapis.com, cloudbilling.googleapis.com, iam.googleapis.com, admin.googleapis.com, and appengine.googleapis.com.

The Project Factory module relies on the google_project_service resource internally to enable the APIs for the projects it creates. This abstracts the complexity of API management, allowing operators to define a set of APIs to enable for all projects in a centralized manner. This approach ensures consistency and reduces the risk of missing a required API enablement in a new project. It also allows for the management of private API concerns by ensuring that only the correct public APIs are enabled, while internal APIs remain managed by Google.

Conclusion

The google_project_service resource is a fundamental component of Terraform configurations for Google Cloud, serving as the bridge between infrastructure code and the underlying service availability. Its proper use requires a clear understanding of the distinction between public and private APIs. Public APIs, such as compute.googleapis.com, are the only valid targets for this resource, while private APIs, such as dataproc-control.googleapis.com, are internal implementation details that do not require customer intervention. Confusing these two categories is a common source of operational errors, leading to failed Terraform runs and unnecessary troubleshooting.

Furthermore, the distinction between API enablement and resource import is vital. API enablement is a prerequisite for resource creation, managed by google_project_service. Resource import is a mechanism for adopting existing resources into Terraform's state, managed by the terraform import command or import blocks. Conflating these concepts leads to misunderstandings about Terraform's capabilities.

For organizations managing multiple projects, the Terraform Project Factory module provides a robust solution for scaling API management. By using opinionated modules and helper scripts, organizations can standardize project creation, ensuring that all necessary APIs are enabled according to best practices. This approach not only reduces manual effort but also enhances security and compliance by enforcing consistent API configurations. In summary, mastering the google_project_service resource involves more than syntactic knowledge; it requires a deep understanding of Google Cloud's API architecture and the strategic use of modules to manage complexity at scale.

Sources

1. Understanding APIs and Terraform
2. terraform-google-project-factory