Cloud Composer is Google Cloud's managed Apache Airflow service, designed to abstract away the operational complexity of orchestrating complex data workflows. In a standard deployment, the service handles the underlying infrastructure for the Airflow web server, scheduler, workers, and metadata database. The primary value proposition for engineering teams is the ability to focus exclusively on writing Directed Acyclic Graphs (DAGs) rather than managing the Kubernetes clusters, database instances, and load balancers required to run them. However, creating a Composer environment through the Google Cloud Console involves a high cardinality of configuration decisions. Practitioners must define networking architectures, scaling parameters, Python package dependencies, and specific Airflow configurations. These manual steps are often difficult to reproduce, leading to configuration drift between development, staging, and production environments. Terraform resolves this by capturing all infrastructure decisions as code. When an organization requires a new environment for testing, staging, or regional failover, applying the same Terraform configuration yields an identical setup, ensuring consistency and auditability across the entire infrastructure lifecycle.

Prerequisites and API Configuration

Before deploying a Cloud Composer environment via Terraform, the target project must have the necessary APIs enabled. Cloud Composer is not a standalone service; it relies on a mesh of underlying Google Cloud Platform services to function correctly. The foundational APIs required for a standard deployment include the Cloud Composer API, Cloud Resource Manager API, Identity and Access Management API, Service Usage API, Cloud Storage API, Compute Engine API, Artifact Registry API, and the Google Kubernetes Engine API. These dependencies exist because Composer utilizes GKE for orchestration, Compute Engine for node provisioning, and Cloud Storage for managing DAG files and logs.

A critical operational consideration when managing these APIs with Terraform is the disable_on_destroy flag. When configuring the google_project_service resource, it is essential to set disable_on_destroy to false for the composer.googleapis.com and container.googleapis.com services. If this flag is omitted or set to true, Terraform will attempt to disable these APIs when the environment is destroyed. This can lead to locking issues or failures if the APIs are still being utilized by other resources or if the service requires a grace period to terminate. Furthermore, some implementations include a check to prevent the disabling of the composer_api through Terraform if an environment existed in the last 30 days, ensuring that accidental destruction does not immediately break active workloads.

Service Account and IAM Best Practices

Proper identity management is a cornerstone of secure Cloud Composer deployments. The service account used for the Composer workers requires specific permissions to interact with the infrastructure. The most critical role is roles/composer.worker, which grants the worker nodes the ability to execute tasks and communicate with the scheduler. Additionally, if the DAGs interact with BigQuery, the service account must be granted roles/bigquery.dataEditor. For DAG files stored in Cloud Storage, the service account requires roles/storage.objectViewer.

Terraform provides the google_service_account resource to create a dedicated service account for the environment. This account is then associated with the necessary roles using the google_project_iam_member resource. It is recommended to use a dedicated service account rather than relying on the default compute engine service account, as this allows for granular permission control and easier auditing. For the service account that provisions the Terraform resources (the CI/CD identity), roles such as roles/composer.admin, roles/iam.serviceAccountUser, and roles/storage.admin are required. The roles/storage.admin permission is specifically needed for Composer to manage the Cloud Storage buckets and objects it creates automatically during environment initialization.

The following Terraform configuration demonstrates the creation of a service account and the assignment of the necessary IAM roles.

```hcl

Service account for Cloud Composer

resource "googleserviceaccount" "composersa" {
accountid = "composer-worker"
displayname = "Cloud Composer Worker"
project = var.projectid
}

Composer workers need the Composer Worker role

resource "googleprojectiammember" "composerworker" {
project = var.projectid
role = "roles/composer.worker"
member = "serviceAccount:${googleserviceaccount.composersa.email}"
}

Grant storage access for DAG files

resource "googleprojectiammember" "storageaccess" {
project = var.projectid
role = "roles/storage.objectViewer"
member = "serviceAccount:${googleserviceaccount.composersa.email}"
}

Grant BigQuery access if DAGs interact with BigQuery

resource "googleprojectiammember" "bigqueryaccess" {
project = var.projectid
role = "roles/bigquery.dataEditor"
member = "serviceAccount:${googleserviceaccount.composersa.email}"
}
```

Versioning and Module Selection

Cloud Composer currently offers two primary versions: V2 and V3. While both provide managed Apache Airflow, V2 is often recommended for new environments due to its greater flexibility in the Airflow core services, such as the scheduler, web server, and workers. V2 offers a more practical and scalable infrastructure architecture. However, V3 provides access to newer Airflow versions and specific features like high resilience mode. The choice between V2 and V3 depends on the specific requirements of the organization, such as the need for high availability or specific Airflow version features.

There are multiple Terraform modules available for managing Cloud Composer. Two prominent modules are terraform-google-modules/composer/google and GoogleCloudPlatform/cloud-composer/google. The terraform-google-modules/composer/google module is designed for Terraform 1.3+ and is currently at version 4.0 or higher, with simple usage examples prioritizing V2 environments. The GoogleCloudPlatform/cloud-composer/google module handles creation and management with specific prerequisites, including Terraform v0.13 and the GCP provider v3.0. The latter module includes specific variables such as image_version, which allows for precise control over the Airflow version, such as composer-3-airflow-2.7.3.

Configuring High Resilience and Scaling

For production workloads, high availability is not optional. Cloud Composer 3 introduces a resilience_mode field within the config block. Setting this field to HIGH_RESILIENCE enables high resilience mode, ensuring that the scheduler and other critical components are replicated across multiple zones. This configuration is particularly important for environments where downtime is unacceptable.

When creating an environment with high resilience, specific scaling parameters must be defined. The --triggerer-cpu and --triggerer-memory flags are required if triggerers are used. Additionally, the --min-workers parameter should be set to 2 or more to ensure that at least two worker nodes are running, providing redundancy for task execution. The gcloud command equivalent to this configuration is:

bash gcloud composer environments create ENVIRONMENT_NAME \ --location LOCATION \ --image-version composer-3-airflow-2.11.1-build.8 \ --service-account "SERVICE_ACCOUNT" \ --enable-high-resilience \ --enable-private-environment \ --scheduler-count 2 \ --triggerer-count 2 \ --triggerer-cpu 0.5 \ --triggerer-memory 0.5 \ --min-workers 2

In Terraform, this configuration is represented using the google_composer_environment resource. The resilience_mode is set to HIGH_RESILIENCE, and the node_config block specifies the service account. It is important to note that for Composer 3 features, the google-beta provider is often required, as these features may not be available in the stable google provider yet.

```hcl
resource "googlecomposerenvironment" "example" {
provider = google-beta
name = "ENVIRONMENT_NAME"
region = "LOCATION"

config {
resiliencemode = "HIGHRESILIENCE"

node_config {
  service_account = "[email protected]"
}

}
}
```

Networking and Private Endpoints

Network isolation is a critical security feature for Cloud Composer environments. By default, Composer environments may have public access to the web server. However, for production environments, it is standard practice to disable public access and use private endpoints. The enable_private_endpoint variable in the Terraform module controls this behavior. When set to false, it configures public access to the cluster endpoint, whereas setting it to true or configuring a private environment restricts access to the VPC network.

For advanced networking configurations, Private Service Connect (PSC) can be used. The composer_network_attachment variable allows for the definition of a PSC network entry point. This is useful for environments that require strict network isolation where the Composer environment is accessed only through a PSC interface within the VPC. The enable_private_environment boolean variable determines if a private Composer environment will be created. When enabled, the environment is isolated from the public internet, and all traffic must flow through the private VPC network. This reduces the attack surface and ensures that sensitive data pipelines are not exposed to external threats.

Advanced Configuration Variables

The flexibility of Terraform modules allows for fine-grained control over the Composer environment. Several key variables facilitate this customization. The airflow_config_overrides variable accepts a map(string) of Apache Airflow configuration properties to override. This allows administrators to modify Airflow's behavior without modifying code, such as adjusting task execution modes or logging levels. The env_variables variable similarly accepts a map(string) to set additional environment variables for the Airflow processes, which is useful for passing secrets or configuration flags to the DAGs.

The environment_size variable determines the size of the environment, with options for ENVIRONMENT_SIZE_SMALL, ENVIRONMENT_SIZE_MEDIUM, and ENVIRONMENT_SIZE_LARGE. This parameter directly impacts the cost and the computational power available to the scheduler and workers. The image_version variable is crucial for controlling the Airflow version, allowing for specific upgrades or downgrades as needed. For example, specifying composer-3-airflow-2.11.1-build.8 ensures that the environment runs a specific build of Airflow 2.11.1.

Conclusion

Deploying Cloud Composer environments with Terraform transforms a complex, manual process into a repeatable, automated infrastructure-as-code workflow. By capturing the decisions regarding networking, scaling, and security in Terraform code, organizations can ensure that their Airflow environments are consistent across all stages of the software development lifecycle. The ability to define high resilience modes, private networking, and specific service account permissions through declarative code reduces the risk of human error and configuration drift. As Cloud Composer evolves, with new features such as V3-specific resilience modes and advanced triggerer configurations, Terraform remains the most robust tool for managing these environments. The integration of modules like terraform-google-modules/composer/google and GoogleCloudPlatform/cloud-composer/google further abstracts the complexity, allowing engineers to focus on the business logic of their DAGs while trusting the infrastructure to be provisioned correctly and securely. The rigorous requirements for API enablement, IAM roles, and network isolation underscore the importance of a well-structured Terraform configuration that adheres to best practices for security and scalability.

Sources

1. How to Create GCP Cloud Composer Environments with Terraform
2. terraform-google-modules/terraform-google-composer
3. GoogleCloudPlatform/terraform-google-cloud-composer
4. Create Composer 3 Environments
5. Create Composer 3 Environments with Terraform