Terraform Driven Cloud Composer Environments

Cloud Composer is Google Cloud's managed Apache Airflow service. It handles the Airflow web server, scheduler, workers, metadata database, and all the infrastructure underneath. You write DAGs, and Composer runs them. But setting up a Composer environment through the console means making dozens of decisions about networking, scaling, Python packages, and Airflow configurations that are hard to reproduce.

Terraform captures all of those decisions in code. When you need a new environment for testing, staging, or a different region, you apply the same configuration and get an identical setup

What Terraform Changes for Composer Operations

Creating a Composer environment via the console requires manual selection of network, subnetwork, service account, image version, node count, and resilience options. Those selections are difficult to version, audit, or reproduce across teams.

Terraform makes the environment definition declarative. The same configuration can be applied to multiple projects and regions. The infrastructure is described in code, so changes to Airflow configuration overrides, environment variables, Python dependencies, scaling parameters, and private networking are all captured in a single plan.

This guide covers creating Cloud Composer 2 environments with Terraform, from basic setups to production configurations with private networking and custom packages.

Module Options for Composer Provisioning

This module makes it easy to create a Cloud Composer Environment. As the module develops, this README should be updated.

The resources/services/activations/deletions that this module will create/trigger are:
- Create a GCP Composer Environment
This module is meant for use with Terraform 1.3+ and tested using Terraform 1.3+. If you find incompatibilities using Terraform >=1.3, please open an issue.
Current version is 4.0. Upgrade guides:
Cloud Composer currently has two versions: V2 has greater flexibility in the Airflow core services (scheduler, webserver, worker) and has a more practical and scalable infrastructure. Therefore, we recommend prioritizing the use of V2 for new environments.

This Terraform module handles the creation and management of Cloud Composer environments on Google Cloud Platform.

This module assumes that the below mentioned prerequisites are in place:
- To deploy this blueprint you must have an active billing account and billing permissions.
- Required APIs are enabled on the project.
- The service account used has the necessary permissions.

Prerequisites and Permissions

These sections describe requirements for using this module.

The following dependencies must be available:
- [Terraform][terraform] v0.13
- [Terraform Provider for GCP][terraform-provider-gcp] plugin v3.0

A service account with the following roles must be used to provision the resources of this module:
- Composer Admin:
roles/composer.admin
- Service Account User:
roles/iam.serviceAccountUser
- Storage Admin:
roles/storage.admin
(Required for Composer to manage GCS buckets and objects) - The service account assigned to var.service_account requires
roles/composer.worker
.

A project with the following APIs enabled must be used to host the resources of this module:
- Cloud Composer API:
composer.googleapis.com
- Cloud Resource Manager API:
cloudresourcemanager.googleapis.com
- IAM API:
iam.googleapis.com
- Service Usage API:
serviceusage.googleapis.com
- Cloud Storage API:
storage.googleapis.com
- Compute Engine API:
compute.googleapis.com
- Artifact Registry API:
artifactregistry.googleapis.com
- Google Kubernetes Engine API:
container.googleapis.com

Enabling Required APIs in Terraform

Cloud Composer depends on several APIs.

Enabling Cloud Composer and its dependencies can be expressed as project services:

```hcl
resource "googleprojectservice" "composer" {
project = var.projectid
service = "composer.googleapis.com"
disableon_destroy = false
}

resource "googleprojectservice" "container" {
project = var.projectid
service = "container.googleapis.com"
disableon_destroy = false
}
```

The module also requires service account creation and IAM bindings before environment creation.

Service Account Setup

Create a dedicated service account for the Composer environment.

hcl resource "google_service_account" "composer_sa" { account_id = "composer-worker" display_name = "Cloud Composer Worker" project = var.project_id }

Composer workers need the Composer Worker role

hcl resource "google_project_iam_member" "composer_worker" { project = var.project_id role = "roles/composer.worker" member = "serviceAccount:${google_service_account.composer_sa.email}" }

Grant storage access for DAG files

hcl resource "google_project_iam_member" "storage_access" { project = var.project_id role = "roles/storage.objectViewer" member = "serviceAccount:${google_service_account.composer_sa.email}" }

Grant BigQuery access if DAGs interact with BigQuery

hcl resource "google_project_iam_member" "bigquery_access" { project = var.project_id role = "roles/bigquery.dataEditor" member =

For custom service accounts used with the GoogleCloudPlatform module:

hcl resource "google_service_account" "custom_service_account" { provider = google-beta account_id = "custom-service-account" display_name = "Example Custom Service Account" }

hcl resource "google_project_iam_member" "custom_service_account" { provider = google-beta project = "example-project" member = format("serviceAccount:%s", google_service_account.custom_service_account.email") role = "roles/composer.worker" }

hcl resource "google_service_account_iam_member" "custom_service_account" { provider = google-beta service_account_id = google_service_account.custom_service_account.name role = "roles/composer.ServiceAgentV2Ext" member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com" }

Module Usage Patterns

Simple usage is as follows:

hcl module "composer" { source = "terraform-google-modules/composer/google" version = "~> 6.4" project_id = "<PROJECT ID>" region = "us-central1" composer_env_name = "composer-env-test" network = "test-network" subnetwork = "composer-subnet" enable_private_endpoint = false }

Functional examples are included in the examples directory.

Basic usage of this module is as follows:

hcl module "cloud_composer_environment" { source = "GoogleCloudPlatform/cloud-composer/google//modules/google-composer-environment" project_id = "<PROJECT ID>" env_name = "<COMPOSER ENV NAME>" region = "<GCP REGION>" image_version = "composer-3-airflow-2.7.3" service_account = "<SERVICE ACCOUNT EMAIL>" }

Functional examples are included in the examples directory.

Module Input Parameters

terraform-google-modules/composer/google parameters:

GoogleCloudPlatform/cloud-composer module parameters:

Environment Configuration with Terraform

When you create an environment, the resilience_mode field in the config block enables the high resilience mode.

hcl resource "google_composer_environment" "example" { provider = google-beta name = "ENVIRONMENT_NAME" region = "LOCATION" config { resilience_mode = "HIGH_RESILIENCE" node_config { service_account =

Example API representation:

json { "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME", "config": { "resilience_mode": "HIGH_RESILIENCE", "nodeConfig": { "serviceAccount": "SERVICE_ACCOUNT" } } }

Example:

json // POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments { "name": "projects/example-project/locations/us-central1/environments/example-environment", "config": { "resilience_mode": "HIGH_RESILIENCE", "nodeConfig": { "serviceAccount": " [email protected] " } } }

Terraform resource for Composer 2 with software config:

hcl resource "google_composer_environment" "example_environment" { provider = google-beta name = "example-environment" config { software_config { image_version = "composer-2.17.5-airflow-2.11.1" } node_config { service_account = google_service_account.custom_service_account.email } } }

Scale and Performance Parameters

If you use triggerers, the--triggerer-cpu and
--triggerer-memory` flags are also required for environment creation.For more information about
--triggerer-count
,--triggerer-cpu
, and--triggerer-memory
flags, see Configure environment scale and performance parameters.--min-workers
to2
or more

Example gcloud command pattern for high resilience:

bash gcloud composer environments create ENVIRONMENT_NAME \ --location LOCATION \ --image-version composer-3-airflow-2.11.1-build.8 \ --service-account "SERVICE_ACCOUNT" \ --enable-high-resilience \ --enable-private-environment \ --scheduler-count 2 \ --triggerer-count 2 \ --triggerer-cpu 0.5 \ --triggerer-memory 0.5 \ --min-workers 2

The same parameters map to Terraform config blocks for software config, node config, and environment config.

Private Networking Options

enableprivateendpoint Configure public access to the cluster endpoint.

enableprivateenvironment If true, a private Composer environment will be created.

composernetworkattachment PSC (Private Service Connect) Network entry point.

network Network where Cloud Composer is created.

subnetwork Composer subnet.

These settings allow the same configuration to be applied to testing, staging, or a different region with identical setup.

Service Usage Protection

To prevent accidental disabling of Composer API when environments exist:

hcl resource "google_project_service" "composer" { //prevent you from disabling composer_api through Terraform if any environment was //there in the last 30 days check_if_service_has_usage_on_destroy = true }

What's next
See other documentation pages for information about configuring your environment with Terraform. For example:
- Create environments
- Override Airflow configuration options
- Set environment variables
- Install Python dependencies
- Scale environments

Conclusion

Terraform provides a reproducible and auditable path to Cloud Composer provisioning. Capturing decisions about networking, scaling, Python packages, and Airflow configurations in code eliminates manual drift between environments. The terraform-google-modules/composer/google module and the GoogleCloudPlatform/cloud-composer module both support Composer V2 with greater flexibility in Airflow core services.

Prerequisites remain consistent: active billing, required APIs enabled, and service accounts with roles/composer.admin, roles/iam.serviceAccountUser, roles/storage.admin, and roles/composer.worker. The service account used to provision must also have Composer Worker role assignment.

Enabling APIs through googleprojectservice resources ensures composer.googleapis.com and container.googleapis.com are active before environment creation. Service account resources combined with googleprojectiam_member bindings provide storage object viewer and BigQuery data editor access where needed.

Configuration depth is achieved through softwareconfig imageversion, nodeconfig serviceaccount, resiliencemode HIGHRESILIENCE, enableprivateenvironment, enableprivateendpoint, network and subnetwork settings, and environment variables. Scale parameters such as scheduler-count, triggerer-count, triggerer-cpu, triggerer-memory, and min-workers translate directly into Terraform config blocks.

Using Terraform for Cloud Composer means the same configuration can be applied to testing, staging, or production regions and produce an identical setup with versioned infrastructure as code.

Sources

1. How to create gcp cloud composer environments with terraform
2. terraform-google-modules/terraform-google-composer
3. GoogleCloudPlatform/terraform-google-cloud-composer
4. Create environments
5. Terraform create environments