Introduction

Cloud Composer is Google Cloud's managed Apache Airflow service, a critical component for data engineers managing complex orchestration workflows at scale. The platform abstracts away the operational burden of managing the Airflow web server, scheduler, workers, and the underlying metadata database, allowing engineers to focus on writing Directed Acyclic Graphs (DAGs). However, the traditional approach of setting up a Composer environment through the Google Cloud Console presents significant challenges for reproducible infrastructure. Console-based provisioning forces administrators to make dozens of granular decisions regarding networking, scaling parameters, Python package dependencies, and Airflow configurations. These decisions are difficult to document accurately and even harder to reproduce across different environments. When a team requires a new environment for testing, staging, or deployment in a different region, the console workflow often leads to configuration drift, where the new environment diverges from the production setup due to manual errors or forgotten settings.

Terraform resolves this challenge by capturing all infrastructure decisions in code. By defining the desired state of the Composer environment in code, teams can apply the same configuration to generate identical setups across any number of environments. This article provides a deep technical analysis of creating Cloud Composer environments using Terraform, covering the transition from Composer 2 to Composer 3, high-resilience configurations, private networking strategies, and the integration of third-party modules for enhanced management.

Understanding the Cloud Composer and Terraform Ecosystem

The integration of Terraform with Cloud Composer is primarily handled through the google_composer_environment resource. This resource allows for the full lifecycle management of the Composer environment, from creation to scaling and deletion. The choice of module or resource implementation can vary based on the specific version of Terraform and the desired level of abstraction.

For teams using the Google Provider for Terraform directly, the google_composer_environment resource provides low-level control. This approach is detailed in the official Google Cloud documentation, which highlights specific fields such as resilience_mode and node_config. When using Composer 3, the API and Terraform configurations have evolved to support higher performance and availability features. For instance, the resilience_mode field in the config block enables high resilience mode, a critical feature for production workloads that require fault tolerance.

In addition to the native provider resources, Google Cloud Platform and the Terraform community offer specialized modules. The terraform-google-composer module, maintained by the Terraform Google Modules repository, is designed to make it easy to create a Cloud Composer Environment. This module is meant for use with Terraform 1.3 and later versions. It abstracts some of the complexity by handling the creation of the GCP Composer Environment as a primary resource. The current version of this module is 4.0, with upgrade guides available for teams transitioning from earlier versions. Another prominent module is the cloud-composer-environment module from the GoogleCloudPlatform repository, which handles the creation and management of Cloud Composer environments on Google Cloud Platform. This module assumes that specific prerequisites are in place, including an active billing account, enabled APIs, and a service account with necessary permissions.

Feature	Native Resource (`google_composer_environment`)	`terraform-google-composer` Module	`GoogleCloudPlatform` Module
Primary Purpose	Low-level API mapping	High-level abstraction	Managed environment setup
Terraform Version	Provider dependent	1.3+	1.0+ (implied)
Key Input	`config` block	`composer_env_name`, `network`	`env_name`, `region`
Resilience Control	Explicit `resilience_mode`	Abstracted	Abstracted
Versioning	Provider specific	Module version (e.g., ~> 6.4)	Module version
Ideal Use Case	Custom high-resilience setups	Standard V2/V3 environments	Complex multi-resource stacks

Prerequisites and API Configuration

Before deploying a Composer environment via Terraform, several prerequisites must be satisfied. First, the project must have an active billing account, as Composer environments incur costs based on node count and duration. Second, the required APIs must be enabled. Cloud Composer depends on several APIs to function correctly, including the Cloud Composer API itself (composer.googleapis.com) and potentially the Container API (container.googleapis.com), although the dependency on the latter has evolved with newer Composer versions.

Terraform can automate the enabling of these APIs using the google_project_service resource. This ensures that the necessary dependencies are active before the environment resource is created. It is critical to set disable_on_destroy to false for these services. This configuration prevents Terraform from attempting to disable the Composer API when the environment is destroyed. If the API is disabled while an environment still exists or if the service is not properly detached, subsequent provisioning steps will fail.

```terraform

Enable Cloud Composer and its dependencies

resource "googleprojectservice" "composer" {
project = var.projectid
service = "composer.googleapis.com"
disableon_destroy = false
}

resource "googleprojectservice" "container" {
project = var.projectid
service = "container.googleapis.com"
disableon_destroy = false
}
```

Additionally, the service account used for deployment must have the necessary permissions to create and manage environments. This typically includes roles such as roles/composer.admin or specific worker roles depending on the granularity of access required.

Service Account and IAM Configuration

A dedicated service account is a best practice for Composer environments. This service account is used by the Airflow workers to execute tasks and access other Google Cloud services. Defining this account in Terraform ensures that the permissions are codified and consistent.

The Composer workers require the roles/composer.worker role to interact with the Composer API. Furthermore, if the DAGs interact with other Google Cloud services, the service account must be granted additional roles. For example, if DAGs read DAG files from Cloud Storage, the roles/storage.objectViewer role is necessary. If DAGs write data to BigQuery, the roles/bigquery.dataEditor role may be required.

```terraform

Service account for Cloud Composer

resource "googleserviceaccount" "composersa" {
accountid = "composer-worker"
displayname = "Cloud Composer Worker"
project = var.projectid
}

Composer workers need the Composer Worker role

resource "googleprojectiammember" "composerworker" {
project = var.projectid
role = "roles/composer.worker"
member = "serviceAccount:${googleserviceaccount.composersa.email}"
}

Grant storage access for DAG files

resource "googleprojectiammember" "storageaccess" {
project = var.projectid
role = "roles/storage.objectViewer"
member = "serviceAccount:${googleserviceaccount.composersa.email}"
}
```

In more complex configurations, such as those using a custom service account defined in a provider block, the role assignment might be specified differently. For instance, when using the google-beta provider, the role for public IP environments is explicitly set to roles/composer.worker.

terraform resource "google_project_iam_member" "custom_service_account" { provider = google-beta project = "example-project" member = format("serviceAccount:%s", google_service_account.custom_service_account.email) role = "roles/composer.worker" }

Creating a High-Resilience Composer 3 Environment

Cloud Composer 3 represents the latest iteration of the service, offering improved performance and scalability. One of the key features of Composer 3 is the ability to configure high-resilience mode. This mode ensures that the environment can withstand failures of individual nodes without impacting the overall availability of the Airflow service.

When creating an environment in high-resilience mode, specific scaling parameters are required. The scheduler_count and triggerer_count must be set appropriately. If triggerers are used, the triggerer-cpu and triggerer-memory flags are also required. Additionally, the min-workers parameter should be set to 2 or more to ensure that worker redundancy is maintained.

The Terraform configuration for a high-resilience Composer 3 environment utilizes the google_composer_environment resource with the provider = google-beta designation, as some of these features may be available only in the beta provider at the time of writing.

terraform resource "google_composer_environment" "example" { provider = google-beta name = "ENVIRONMENT_NAME" region = "LOCATION" config { resilience_mode = "HIGH_RESILIENCE" node_config { service_account = "[email protected]" } } }

The image version is a critical parameter in Composer 3. A typical image version format is composer-3-airflow-2.11.1-build.8. Specifying the exact image version ensures that the environment is built with the precise Airflow version and Composer configuration required for the workload.

terraform resource "google_composer_environment" "example_environment" { provider = google-beta name = "example-environment" config { software_config { image_version = "composer-3-airflow-2.11.1-build.8" } node_config { service_account = google_service_account.custom_service_account.email } } }

Networking and Private Environments

Networking is a central concern in Composer environment design. By default, Composer environments may have public endpoints, but for production security, private environments are preferred. A private Composer environment creates a private VPC network where all traffic between the Airflow components and external services is routed privately.

The enable_private_environment flag in the module configurations controls this behavior. When set to true, a private Composer environment will be created. The composer_network_attachment parameter is used to specify the Private Service Connect (PSC) Network entry point, which allows the Composer environment to securely consume other services.

For teams using the terraform-google-composer module, the networking configuration is abstracted through inputs like network and subnetwork.

terraform module "composer" { source = "terraform-google-modules/composer/google" version = "~> 6.4" project_id = "<PROJECT ID>" region = "us-central1" composer_env_name = "composer-env-test" network = "test-network" subnetwork = "composer-subnet" enable_private_endpoint = false }

The enable_private_endpoint parameter configures public access to the cluster endpoint. In a production scenario, this would typically be set to false to enforce private connectivity.

Connecting external services, such as Cloud SQL, to a private Composer environment requires specific networking setups. One common pattern is using the Cloud SQL Proxy. This proxy allows clients to connect to Cloud SQL instances without requiring the client to be in the same VPC, or by establishing a Private Service Connect connection. The setup involves downloading the proxy binary, configuring the network, and using gcloud commands or Terraform to create forwarding rules and DNS zones.

bash URL="https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.13.0" curl "$URL/cloud-sql-proxy.linux.amd64" -o cloud-sql-proxy chmod +x cloud-sql-proxy

The networking for the proxy often involves creating a private DNS zone for the Cloud SQL instance.

```bash
gcloud dns managed-zones create cloud-sql-dns-zone \
--project= \
--description="DNS zone for the Cloud SQL instance" \
--dns-name= \
--networks=nw1-vpc \
--visibility=private

gcloud dns record-sets create \
--project= \
--type=A \
--rrdatas=10.10.1.10 \
--zone=cloud-sql-dns-zone
```

Module Inputs and Configuration Details

Both the terraform-google-composer and GoogleCloudPlatform modules provide a range of input variables to customize the environment. Understanding these inputs is essential for fine-tuning the deployment.

terraform-google-composer Module Inputs

Name	Description	Type	Default	Required
`composer_env_name`	Name of Cloud Composer Environment	string	n/a	yes
`enable_private_endpoint`	Configure public access to the cluster endpoint.	bool	false	no
`network`	Network where Cloud Composer is created.	string	n/a	yes
`project_id`	Project ID where Cloud Composer Environment is created	string	n/a	yes
`subnetwork`	Subnetwork where Cloud Composer is created.	string	n/a	yes

GoogleCloudPlatform cloud-composer-environment Module Inputs

Name	Description	Type	Default	Required
`airflow_config_overrides`	Apache Airflow configuration properties to override.	map(string)	{}	no
`composer_network_attachment`	PSC (Private Service Connect) Network entry point.	string	null	no
`enable_private_environment`	If true, a private Composer environment will be created.	bool	false	no
`env_name`	Name of the Composer environment.	string	n/a	yes
`env_variables`	Additional environment variables for the Airflow processes.	map(string)	{}	no
`environment_size`	The environment size (ENVIRONMENTSIZESMALL, ENVIRONMENTSIZEMEDIUM, ENVIRONMENTSIZELARGE)	string	n/a	yes
`image_version`	The image version for the Composer environment.	string	n/a	no
`service_account`	The service account used by the Composer environment.	string	n/a	yes

The airflow_config_overrides variable allows for deep customization of the Airflow engine, enabling teams to tune scheduler intervals, web server ports, or other core parameters. The env_variables variable injects additional environment variables into the Airflow processes, useful for configuration management or secret injection.

Best Practices for Production Environments

When moving from a simple test environment to a production setup, several best practices should be adhered to.

Versioning and Upgrading: The terraform-google-composer module notes that as it develops, the README should be updated. Teams should pin their module versions to avoid unexpected breaking changes during upgrades. The module is tested using Terraform 1.3+, and any incompatibilities with Terraform versions greater than 1.3 should be reported as issues to the maintainers.
Resilience and Scaling: For production workloads, high-resilience mode is strongly recommended. This requires configuring the scheduler and triggerer counts appropriately. Setting min-workers to 2 or more ensures that worker failures do not cause task backlogs.
Security: Private environments should be the default for production. Access to the Airflow web interface should be restricted via IP allowlists or private service connect. The service account permissions should follow the principle of least privilege, granting only the necessary roles for the specific DAGs being executed.
Cost Management: The environment_size parameter in the GoogleCloudPlatform module allows for scaling the environment. Using ENVIRONMENT_SIZE_SMALL for development and ENVIRONMENT_SIZE_LARGE for production can help optimize costs. Additionally, automated scaling policies can be configured to reduce costs during off-peak hours.
Disaster Recovery: The check_if_service_has_usage_on_destroy parameter in some configurations prevents the disabling of the Composer API if an environment was present in the last 30 days. This is a safety mechanism to prevent accidental loss of data or configuration during cleanup operations.

terraform resource "google_project_service" "composer" { project = var.project_id service = "composer.googleapis.com" disable_on_destroy = false # Prevents disabling if usage exists in last 30 days check_if_service_has_usage_on_destroy = true }

Conclusion

Creating Cloud Composer environments with Terraform transforms the deployment process from a manual, error-prone task into a deterministic, code-driven workflow. By leveraging the google_composer_environment resource and specialized modules like terraform-google-composer and GoogleCloudPlatform/cloud-composer, engineers can manage complex infrastructure configurations with precision. The transition to Composer 3 introduces new capabilities such as high-resilience modes and updated image versions, which must be carefully configured in Terraform to take full advantage of the service's improved performance.

The depth of control provided by Terraform allows for the automation of API enablement, service account creation, IAM role assignment, and networking setup. Whether using native provider resources for granular control over resilience modes or high-level modules for streamlined deployment, the choice depends on the team's specific requirements and operational maturity. As Cloud Composer continues to evolve, staying updated with module versions and Terraform compatibility is essential. The integration of private networking, Cloud SQL proxies, and custom Airflow configurations through Terraform ensures that data engineering teams can build secure, scalable, and reproducible orchestration environments that meet the demands of modern data pipelines.

Engineering Cloud Composer Environments with Terraform: A Comprehensive Guide