Google Cloud Pub/Sub is one of those services that shows up in almost every GCP architecture. Whether you are building event-driven microservices, streaming data into BigQuery, or triggering Cloud Functions, Pub/Sub is usually the glue in between. And if you are managing your infrastructure with Terraform, you will want to define your topics and subscriptions as code rather than clicking through the Console.
This guide walks through creating Pub/Sub topics and subscriptions with Terraform, covering the common configurations you will need in a real project.
Provider Configuration
Before creating any Pub/Sub resources, make sure your Terraform configuration includes the Google provider.
hcl
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0"
}
}
}
The provider block ties the configuration to a specific project and region.
hcl
provider "google" {
project = var.project_id
region = var.region
}
Variables for project and region are defined explicitly.
hcl
variable "project_id" {
description = "The GCP project ID"
type = string
}
variable "region" {
description = "The GCP region"
type = string
default = "us-central1"
}
Creating a Basic Topic and Subscription
The simplest Pub/Sub setup is a topic with a pull subscription.
hcl
resource "google_pubsub_topic" "topics" {
for_each = var.pubsub_config
name = each.key
project = var.project_id
message_retention_duration = "604800s"
}
message_retention_duration is set to 604800s to keep messages for 7 days.
Subscriptions are created for every topic using a local flatten of the configuration.
hcl
variable "pubsub_config" {
type = map(object({
subscriptions = list(object({
name = string
subscribers = list(string)
}))
publishers = list(string)
}))
description = "Pub/Sub configuration, use topics name as key"
}
hcl
locals {
subscriptions = flatten([
for topic, config in var.pubsub_config : [
for sub in config.subscriptions : {
topic = topic
name = sub.name
subscribers = sub.subscribers
} if length(config.subscriptions) > 0
]
])
}
The subscription resource uses a tomap for_each keyed on topic and subscription name.
hcl
resource "google_pubsub_subscription" "subscriptions" {
for_each = tomap({
for subscription in local.subscriptions : "${subscription.topic}-${subscription.name}-subscription" => {
name = subscription.name
topic = subscription.topic
}
})
name = each.value.name
topic = google_pubsub_topic.topics[each.value.topic].name
project = var.project_id
ack_deadline_seconds = 600
}
Use pull subscriptions for worker-based processing and push for HTTP webhook delivery. Always configure dead-letter topics, set appropriate retry policies, and use message filtering to route events efficiently. BigQuery subscriptions give you analytics without writing any code.
IAM Permissions and Publisher Roles
Granting publisher roles to service accounts is common for event-driven architectures.
hcl
resource "google_pubsub_topic_iam_binding" "publisher"
A specific example for a dead-letter queue publisher is:
hcl
resource "google_pubsub_topic_iam_member" "dlq_publisher" {
topic = google_pubsub_topic.order_events_dlq.name
role = "roles/pubsub.publisher"
member = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
Subscriber permissions for the Pub/Sub service account can be granted as:
hcl
resource "google_pubsub_subscription_iam_member" "dlq_subscriber" {
subscription = google_pubsub_subscription.order_processor_with_dlq.name
role = "roles/pubsub.subscriber"
member = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
The project number is retrieved via:
hcl
data "google_project" "current" {}
Module Patterns for Reuse
If you have many topics and subscriptions that follow similar patterns, wrap them in a Terraform module.
hcl
variable "topic_name" {
type = string
}
variable "subscriptions" {
type = map(object({
ack_deadline = number
max_delivery_attempts = number
}))
}
variable "labels" {
type = map(string)
default = {}
}
A reusable module layout:
hcl
data "google_project" "current" {}
locals {
pubsub_service_account = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
resource "google_pubsub_topic" "main" {
name = var.topic_name
labels = var.labels
message_retention_duration = "604800s"
}
resource "google_pubsub_topic" "dlq"
This module makes it easy to create Google Cloud Pub/Sub topic and subscriptions associated with the topic. This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue. If you haven't upgraded and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is v1.9.0. This is a simple usage of the module.
Create a Terraform module to create GCP pubsub resources, including topics, subscriptions and iam permissions. Terraform code first define a Terraform variable pubsub_config to store your Pub/Sub configuration, then extract your subscriptions to a local variable.
Dead Letter Queues and Resilience
Dead letter topics are essential for resilience. The configuration parameters are based on the terraform-google-pubsub Terraform module.
Grant Pub/Sub service account permission to publish to the DLQ topic and permission to subscribe on the source subscription as shown above.
Deploy Google Cloud Functions with Terraform. HTTP and event-driven functions, Pub/Sub triggers, Cloud Storage triggers, VPC connectors, and IAM configuration.
Deploy Google Cloud Run services with Terraform. Container deployment, custom domains, traffic splitting, IAM, and VPC connector configuration.
Deploy GCP Cloud SQL instances with Terraform. MySQL and PostgreSQL configuration, private networking, backups, replicas, and database user management.
Deploy Google Kubernetes Engine clusters with Terraform. Autopilot and Standard modes, node pools, networking, and workload identity.
Common Pitfalls
A few things that trip people up when managing Pub/Sub with Terraform:
- Subscription expiration: By default, subscriptions expire after 31 days of inactivity. If your subscription seems to disappear, set
hcl
expiration_policy { ttl = "" }
to disable expiration.
Changing topic on a subscription: You cannot change the topic of an existing subscription. Terraform will destroy and recreate it, which means you lose any unprocessed messages.
IAM propagation delays: After granting IAM roles, it can take a few minutes for the permissions to propagate. If dead letter publishing fails immediately after creation, wait and retry.
Message ordering: If you need ordered delivery, you must set
hcl
enable_message_ordering = true
on the subscription. This cannot be changed after creation without recreating the subscription.
Adjust your Terraform code until the plan shows no changes.
Component Connections
The following table includes the components that you can connect to a Pub/Sub topic or subscription, and the resulting updates to your application and its generated Terraform code.
| Connected component | Application updates | Background information |
|---|---|---|
| Compute Engine instance template | The Compute Engine instances can publish to and receive messages from the Pub/Sub topic. The Pub/Sub topic ID is added to the Compute Engine instance template metadata. | Instance templates |
| Service account | The service account can manage Pub/Sub topics, and pull messages from subscriptions. The roles/pubsub.editor role is added to the service account. The service account name and email information is added to the Pub/Sub pull subscription. | Access control with IAM |
| BigQuery | The Pub/Sub subscription can write messages to the BigQuery dataset. The BigQuery dataset information is added to the BigQuery subscription fields. | BigQuery subscriptions |
| Cloud Run | The Cloud Run service can receive messages or publish to the Pub/Sub topic. The Pub/Sub topic ID is added to the Cloud Run environment variables. The roles/pubsub.publisher and roles/pubsub.subscriber roles are added to the Cloud Run service account. The Cloud Run service is added to the Pub/Sub push and pull subscription fields. | Use Pub/Sub with Cloud Run tutorial |
| Cloud Storage | The Pub/Sub subscription can write messages to the Cloud Storage |
Conclusion
Terraform is the right way to manage Pub/Sub infrastructure in any serious GCP project. It gives you version control, peer review, and reproducibility for your messaging infrastructure. Start with simple topics and pull subscriptions, add dead letter topics for resilience, and use modules when patterns start repeating. The initial investment in writing Terraform code pays for itself the first time you need to replicate your setup in a new environment.
The combination of provider configuration, variable-driven topic definitions, local subscription flattening, IAM bindings for publishers and subscribers, and module encapsulation provides a complete as-code workflow for Pub/Sub. Pairing this with awareness of expiration policies, immutable topic bindings on subscriptions, IAM propagation delays, and message ordering requirements avoids the most common operational issues. Using pull subscriptions for worker-based processing and push for HTTP webhook delivery, with dead-letter topics and retry policies, creates durable event-driven architectures that integrate cleanly with Cloud Run, Cloud Functions, BigQuery, Compute Engine and Cloud Storage.