Cloud Composer is Google Cloud's managed Apache Airflow service. It handles the Airflow web server, scheduler, workers, metadata database, and all the infrastructure underneath. You write DAGs, and Composer runs them. But setting up a Composer environment through the console means making dozens of decisions about networking, scaling, Python packages, and Airflow configurations that are hard to reproduce. Terraform captures all of those decisions in code. When you need a new environment for testing, staging, or a different region, you apply the same configuration and get an identical setup.
This guide covers creating Cloud Composer 2 environments with Terraform, from basic setups to production configurations with private networking and custom packages.
Why Terraform for Cloud Composer
Infrastructure as code for Composer solves reproducibility and drift. Console provisioning hides the interplay between software image version, node configuration, service account permissions, required APIs, and network settings. Terraform makes those decisions explicit, version controlled, and repeatable across projects.
The guide covers creating Cloud Composer 2 environments with Terraform, from basic setups to production configurations with private networking and custom packages.
Prerequisites and Required APIs
This Terraform module handles the creation and management of Cloud Composer environments on Google Cloud Platform. This module assumes that the below mentioned prerequisites are in place:
- To deploy this blueprint you must have an active billing account and billing permissions.
- Required APIs are enabled on the project.
- The service account used has the necessary permissions.
Cloud Composer depends on several APIs. Enabling them is typically done with google_project_service.
resource "google_project_service" "composer" {
project = var.project_id
service = "composer.googleapis.com"
disable_on_destroy = false
}
resource "google_project_service" "container" {
project = var.project_id
service = "container.googleapis.com"
disable_on_destroy = false
}
Before running terraform apply, service accounts that must exist are referenced in practice:
${project_number}[email protected]service-${project_number}@cloudcomposer-accounts.iam.gserviceaccount.com
The workflow is:
terraform initterraform planterraform apply
After these steps, you can create Cloud Composer via Terraform.
Core Modules and Options
Two public Terraform modules are commonly used for Composer.
The GoogleCloudPlatform module.
This Terraform module handles the creation and management of Cloud Composer environments on Google Cloud Platform.
Basic usage of this module is as follows:
module "cloud_composer_environment" {
source = "GoogleCloudPlatform/cloud-composer/google//modules/google-composer-environment"
project_id = "<PROJECT ID>"
env_name = "<COMPOSER ENV NAME>"
region = "<GCP REGION>"
image_version = "composer-3-airflow-2.7.3"
service_account = "<SERVICE ACCOUNT EMAIL>"
}
Functional examples are included in the examples directory.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| airflowconfigoverrides | Apache Airflow configuration properties to override. | map(string) | {} | no |
| composernetworkattachment | PSC (Private Service Connect) Network entry point. | string | null | no |
| enableprivateenvironment | If true, a private Composer environment will be created. | bool | false | no |
| env_name | Name of the Composer environment. | string | n/a | yes |
| env_variables | Additional environment variables for the Airflow processes. | map(string) | {} | no |
| environment_size | The environment size (ENVIRONMENTSIZESMALL, ENVIRONMENTSIZEMEDIUM, ENVIRONMENTSIZELARGE) |
The terraform-google-modules module.
This module makes it easy to create a Cloud Composer Environment. As the module develops, this README should be updated.
The resources/services/activations/deletions that this module will create/trigger are:
- Create a GCP Composer Environment
This module is meant for use with Terraform 1.3+ and tested using Terraform 1.3+. If you find incompatibilities using Terraform >=1.3, please open an issue.
Current version is 4.0. Upgrade guides:
Cloud Composer currently has two versions: V2 has greater flexibility in the Airflow core services (scheduler, webserver, worker) and has a more practical and scalable infrastructure. Therefore, we recommend prioritizing the use of V2 for new environments.
Simple usage is as follows:
module "composer" {
source = "terraform-google-modules/composer/google"
version = "~> 6.4"
project_id = "<PROJECT ID>"
region = "us-central1"
composer_env_name = "composer-env-test"
network = "test-network"
subnetwork = "composer-subnet"
enable_private_endpoint = false
}
Functional examples are included in the examples directory.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| composerenvname | Name of Cloud Composer Environment | string | n/a | yes |
| enableprivateendpoint | Configure public access to the cluster endpoint. | bool | false | no |
| network | Network where Cloud Composer is created. | string | n/a | yes |
| project_id | Project ID where Cloud Composer Environment is created |
Service Account and IAM Configuration
Create a dedicated service account for the Composer environment.
resource "google_service_account" "composer_sa" {
account_id = "composer-worker"
display_name = "Cloud Composer Worker"
project = var.project_id
}
Composer workers need the Composer Worker role.
resource "google_project_iam_member" "composer_worker" {
project = var.project_id
role = "roles/composer.worker"
member = "serviceAccount:${google_service_account.composer_sa.email}"
}
Grant storage access for DAG files.
resource "google_project_iam_member" "storage_access" {
project = var.project_id
role = "roles/storage.objectViewer"
member = "serviceAccount:${google_service_account.composer_sa.email}"
}
Grant BigQuery access if DAGs interact with BigQuery.
resource "google_project_iam_member" "bigquery_access" {
project = var.project_id
role = "roles/bigquery.dataEditor"
member =
An alternative custom service account pattern is shown with provider google-beta.
resource "google_service_account" "custom_service_account" {
provider = google-beta
account_id = "custom-service-account"
display_name = "Example Custom Service Account"
}
resource "google_project_iam_member" "custom_service_account" {
provider = google-beta
project = "example-project"
member = format("serviceAccount:%s", google_service_account.custom_service_account.email)
// Role for Public IP environments
role = "roles/composer.worker"
}
resource "google_service_account_iam_member" "custom_service_account" {
provider = google-beta
service_account_id = google_service_account.custom_service_account.name
role = "roles/composer.ServiceAgentV2Ext"
member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
}
Environment Configuration and Versioning
This article introduces how to build Cloud Composer using Terraform. Currently, both Version 1 and Version 2 of Cloud Composer exist. In this article, we will use Version 2 for the setup.
This article is primarily intended for the following readers:
If you are not familiar with how to use Terraform, please refer to this article.
Let's get started with the Terraform configuration.
Skipping the basic settings such as the provider configuration, here we list only the files necessary for setting up Cloud Composer.
First, the variable definitions include the Cloud Composer version and the workload configuration. These settings affect the cost of Cloud Composer, so for small-scale data pipeline setups, large specifications are not necessary.
A concrete environment definition using google-beta is:
resource "google_composer_environment" "example_environment" {
provider = google-beta
name = "example-environment"
config {
software_config {
image_version = "composer-2.17.5-airflow-2.11.1"
}
node_config {
service_account = google_service_account.custom_service_account.email
}
}
}
What's next
See other documentation pages for information about configuring your environment with Terraform. For example:
- Create environments
- Override Airflow configuration options
- Set environment variables
- Install Python dependencies
- Scale environments
Service protection on destroy can be set as:
check_if_service_has_usage_on_destroy = true
}
If set to true it will
//prevent you from disabling composer_api through Terraform if any environment was
//there in the last 30 days
Private Networking and Security
Private environments are a key production requirement.
The parameter enableprivateenvironment controls whether a private Composer environment will be created. The module parameter enableprivateendpoint configures public access to the cluster endpoint.
The composernetworkattachment parameter provides PSC (Private Service Connect) Network entry point.
Composer V2 has greater flexibility in the Airflow core services (scheduler, webserver, worker) and has a more practical and scalable infrastructure. Therefore, we recommend prioritizing the use of V2 for new environments.
Cost Considerations and Workload Sizing
Variable definitions include the Cloud Composer version and the workload configuration. These settings affect the cost of Cloud Composer, so for small-scale data pipeline setups, large specifications are not necessary.
Environment size options are ENVIRONMENTSIZESMALL, ENVIRONMENTSIZEMEDIUM, ENVIRONMENTSIZELARGE. Airflow config overrides and environment variables allow tuning without increasing machine size.
Conclusion
Terraform provides a repeatable, auditable way to provision Cloud Composer 2 environments with consistent networking, IAM, software versions, and sizing. Using the GoogleCloudPlatform module or the terraform-google-modules module reduces boilerplate while exposing the parameters that matter for production: project and region, environment name, network and subnetwork, private endpoint, service account, image version, Airflow config overrides, and environment variables.
Prerequisites remain critical: an active billing account, required APIs enabled, and correctly scoped service accounts with roles such as roles/composer.worker, roles/storage.objectViewer, roles/bigquery.dataEditor, and roles/composer.ServiceAgentV2Ext. Enabling composer.googleapis.com and container.googleapis.com via googleprojectservice ensures the API is available before environment creation.
For new environments, prioritize Cloud Composer V2 with explicit image_version such as composer-2.17.5-airflow-2.11.1 or composer-3-airflow-2.7.3, and define workload sizing with care. The module examples provide a safe starting point, and the variable tables make it straightforward to extend configurations for private networking, PSC attachments, and custom Python packages while keeping all decisions in code.