Terraform PubSub Subscriptions: Topics, Dead-Letter Queues, and Reusable Modules for GCP

Google Cloud Pub/Sub is one of those services that shows up in almost every GCP architecture. Whether you are building event-driven microservices, streaming data into BigQuery, or triggering Cloud Functions, Pub/Sub is usually the glue in between. And if you are managing your infrastructure with Terraform, you will want to define your topics and subscriptions as code rather than clicking through the Console.

This guide walks through creating Pub/Sub topics and subscriptions with Terraform, covering the common configurations you will need in a real project.

Setting Up the Provider

Before creating any Pub/Sub resources, make sure your Terraform configuration includes the Google provider:

```hcl
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0"
}
}
}

provider "google" {
project = var.project_id
region = var.region
}

variable "project_id" {
description = "The GCP project ID"
type = string
}

variable "region" {
description = "The GCP region"
type = string
default = "us-central1"
}
```

The provider configuration pins the project and region used for all subsequent Pub/Sub resources. Variable definitions allow reuse across environments.

Creating a Basic Topic and Pull Subscription

The simplest Pub/Sub setup is a topic with a pull subscription. Terraform can handle this:

hcl resource "google_pubsub_topic" "main" { name = var.topic_name labels = var.labels message_retention_duration = "604800s" }

The messageretentionduration is set to 604800s which corresponds to 7 days retention. This is a common baseline for event-driven workloads.

A reusable module for topic with DLQ illustrates the pattern:

```hcl
variable "topic_name" {
type = string
}

variable "subscriptions" {
type = map(object({
ackdeadline = number
maxdelivery_attempts = number
}))
}

variable "labels" {
type = map(string)
default = {}
}

data "google_project" "current" {}

locals {
pubsubserviceaccount = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "googlepubsubtopic" "main" {
name = var.topicname
labels = var.labels
messageretention_duration = "604800s"
}

resource "googlepubsubtopic" "dlq"
```

Dead-letter queue handling requires IAM grants for the Pub/Sub service account to publish to the DLQ topic and subscribe on the source subscription.

```hcl
resource "googlepubsubtopiciammember" "dlqpublisher" {
topic = googlepubsubtopic.ordereventsdlq.name
role = "roles/pubsub.publisher"
member = "serviceAccount:service-${data.googleproject.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "googlepubsubsubscriptioniammember" "dlqsubscriber" {
subscription = googlepubsubsubscription.orderprocessorwithdlq.name
role = "roles/pubsub.subscriber"
member = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

data "google_project" "current" {}
```

These IAM members ensure the managed Pub/Sub service account can move failed messages to the dead-letter topic and read from the source subscription.

Module-Based Configuration with Dynamic Subscriptions

If you have many topics and subscriptions that follow similar patterns, wrap them in a Terraform module.

A common pattern defines a variable to store Pub/Sub configuration, then extract subscriptions to a local variable.

```hcl
variable "pubsub_config" {
type = map(object({
subscriptions = list(object({
name = string
subscribers = list(string)
}))
publishers = list(string)
}))
description = "Pub/Sub configuration, use topics name as key"
}

locals {
subscriptions = flatten([
for topic, config in var.pubsub_config : [
for sub in config.subscriptions : {
topic = topic
name = sub.name
subscribers = sub.subscribers
} if length(config.subscriptions) > 0
]
])
}
```

Create Pub/Sub Topics:

hcl resource "google_pubsub_topic" "topics" { for_each = var.pubsub_config name = each.key project = var.project_id message_retention_duration = "604800s" }

Create subscriptions for every topic:

```hcl
resource "googlepubsubsubscription" "subscriptions" {
for_each = tomap({
for subscription in local.subscriptions : "${subscription.topic}-${subscription.name}-subscription" => {
name = subscription.name
topic = subscription.topic
}
})

name = each.value.name
topic = googlepubsubtopic.topics[each.value.topic].name
project = var.projectid
ackdeadline_seconds = 600
}
```

Grant Topic Publisher role to GSA:

hcl resource "google_pubsub_topic_iam_binding"

This approach centralizes configuration and makes it easy to add new topics by extending var.pubsub_config.

Using the Terraform Google Modules PubSub Module

The terraform-google-modules/terraform-google-pubsub module makes it easy to create Google Cloud Pub/Sub topic and subscriptions associated with the topic. This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue. If you haven't upgraded and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is v1.9.0.

A simple usage of the module:

```hcl
module "pubsub" {
source = "terraform-google-modules/pubsub/google"
version = "~> 8.7"
topic = "tf-topic"
project_id = "my-pubsub-project"

pushsubscriptions = [
{
name = "push"
ackdeadlineseconds = 20
pushendpoint = "https://example.com"
x-goog-version = "v1beta1"
oidcserviceaccountemail = "[email protected]"
audience = "example"
expirationpolicy = "1209600s"
deadlettertopic = "projects/my-pubsub-project/topics/example-dl-topic"
maxdeliveryattempts = 5
maximumbackoff = "600s"
minimumbackoff = "300s"
filter = "attributes.domain = \"com\""
enablemessageordering = true
}
]

pullsubscriptions = [
{
name = "pull"
ackdeadlineseconds = 20
deadlettertopic = "projects/my-pubsub-project/topics/example-dl-topic"
maxdeliveryattempts = 5
maximumbackoff = "600s"
minimumbackoff = "300s"
filter = "attributes.domain = \"com\""
enablemessageordering = true
serviceaccount = "[email protected]"
enableexactlyonce_delivery = true
}
]

bigquerysubscriptions = [
{
name = "bigquery"
table = "project.dataset.table"
usetopicschema = true
usetable_schema = false
}
]
}
```

The module supports push subscriptions, pull subscriptions, and BigQuery subscriptions with options for dead-letter topics, backoff, filtering, and message ordering.

Component Connections to Pub/Sub

The configuration parameters are based on the terraform-google-pubsub Terraform module.

The following table includes the components that you can connect to a Pub/Sub topic or subscription, and the resulting updates to your application and its generated Terraform code.

This mapping shows how Terraform wiring changes when Pub/Sub is connected to Compute Engine, Service Accounts, BigQuery, Cloud Run, and Cloud Storage.

Common Pitfalls When Managing Pub/Sub with Terraform

A few things that trip people up when managing Pub/Sub with Terraform:

• Subscription expiration: By default, subscriptions expire after 31 days of inactivity. If your subscription seems to disappear, set
  expiration_policy { ttl = "" }
  to disable expiration.

• Changing topic on a subscription: You cannot change the topic of an existing subscription. Terraform will destroy and recreate it, which means you lose any unprocessed messages.

• IAM propagation delays: After granting IAM roles, it can take a few minutes for the permissions to propagate. If dead letter publishing fails immediately after creation, wait and retry.

• Message ordering: If you need ordered delivery, you must set
  enable_message_ordering = true
  on the subscription. This cannot be changed after creation without recreating the subscription.

Adjust your Terraform code until the plan shows no changes.

Wrapping Up

Terraform is the right way to manage Pub/Sub infrastructure in any serious GCP project. It gives you version control, peer review, and reproducibility for your messaging infrastructure. Start with simple topics and pull subscriptions, add dead letter topics for resilience, and use modules when patterns start repeating. The initial investment in writing Terraform code pays for itself the first time you need to replicate your setup in a new environment.

Conclusion

Creating Pub/Sub subscriptions with Terraform requires precise handling of topics, subscriptions, IAM, and lifecycle constraints. Provider setup establishes the project and region context for all resources. Basic topic creation with message retention of 604800s provides a 7-day window for consumers. Dead-letter queues add resilience, but require explicit IAM grants for the Pub/Sub service account to publish and subscribe.

Module-based patterns using foreach and local flattening allow a single pubsubconfig variable to drive many topics and subscriptions, reducing duplication. The terraform-google-modules/terraform-google-pubsub module abstracts push, pull, and BigQuery subscription options with parameters for ackdeadlineseconds, maxdeliveryattempts, backoff, filtering, and enablemessageordering.

Component connections demonstrate how Pub/Sub integrates with Compute Engine instance templates, service accounts, BigQuery, Cloud Run, and Cloud Storage, each triggering specific Terraform updates to metadata, IAM roles, environment variables, and subscription fields.

Operational pitfalls remain critical: subscription expiration after 31 days of inactivity, immutable topic-to-subscription binding, IAM propagation delays, and immutable enablemessageordering once created. Handling these constraints in code prevents data loss and failed deliveries.

Together, declarative Terraform definitions, reusable modules, and awareness of Pub/Sub lifecycle rules enable reliable, repeatable messaging infrastructure across GCP projects.

Sources

1. OneUptime Blog
2. AmyInfo Blog
3. Google Cloud Docs
4. Terraform Google Modules GitHub