Serverless event processing in AWS is built around decoupled queues and just-in-time compute. The combination of Amazon Simple Queue Service and AWS Lambda is a common production pattern for reliable asynchronous workloads. When the infrastructure is defined with Terraform, the queue, the function, the IAM permissions, and the event source mapping can be versioned, reviewed, and reproduced across environments.
This article covers the reference architecture for invoking a Lambda function when a message arrives in an SQS queue, the Terraform resources required to implement it, and the operational settings that keep the pipeline reliable. The discussion is grounded in concrete Terraform examples that create an SQS queue with a Lambda trigger, configure dead-letter handling, set visibility and retention parameters, and provision least-privilege IAM roles for queue access.
Core Services Overview
Lambda functions are a serverless service in AWS where you can run code without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.
Amazon Simple Queue Service (Amazon SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. The service enables you to integrate and decouple microservices with throttling.
The simple solution architecture will invoke a lambda function when a message arrives in our SQS queue. The lambda function will load code from a S3 bucket and will print a message.
Objectives of a Complete Terraform Template
From the reference implementations, you will be able to:
- Create a Lambda IAM role.
- Deploy a Lambda function using Terraform.
- Create a SNS topic and set this as a trigger for your lambda function.
- Create a S3 bucket to store all lambda function code.
The template includes all necessary IAM permissions and resource configurations.
Standard queue with configurable visibility timeout and message retention
Node.js function with SQS trigger and proper IAM permissions
Least privilege permissions for Lambda to consume SQS messages
Terraform Provider and Provider Configuration
A complete Terraform configuration to deploy an SQS queue with a Lambda function trigger starts with the provider block.
hcl
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
The template creates an AWS SQS queue and configures a Lambda function to be triggered by messages in the queue.
SQS Queue Definition
Create SQS queue
hcl
resource "aws_sqs_queue" "example_queue" {
name = "example-queue"
delay_seconds = 0
max_message_size = 262144
message_retention_seconds = 345600
receive_wait_time_seconds = 10
visibility_timeout_seconds = 30
tags = {
Environment = "production"
}
}
For order processing workloads, a more detailed queue configuration is used:
hcl
resource "aws_sqs_queue" "orders" {
name = "order-processing-queue"
visibility_timeout_seconds = 1805
message_retention_seconds = 1209600
receive_wait_time_seconds = 20
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.orders_dlq.arn
maxReceiveCount = 3
})
tags = {
Name = "order-processing-queue"
}
}
Key parameters from the reference:
- How long a message is hidden after being received
- Must be >= Lambda timeout; AWS recommends at least 6x the timeout plus batching window
- How long messages stay in the queue before being deleted
- Long polling - reduces empty receives and cost
- Server-side encryption
The visibility timeout must be set to at least six times the function timeout. This ensures Lambda has enough time to retry if a function is throttled while processing a previous batch.
Dead letter queue configuration is applied via redrive_policy.
hcl
resource "aws_sqs_queue" "orders_dlq" {
name = "order-processing-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
tags = {
Name = "order-processing-dlq"
}
}
Dead letter queue for failed messages
Keep failed messages for 14 days
Lambda Function Definition
Lambda function that processes messages
hcl
resource "aws_lambda_function" "order_processor" {
function_name = "order-processor"
handler = "index.handler"
runtime = "python3.12"
role = aws_iam_role.lambda_exec.arn
timeout = 300
}
The timeout of 300 seconds is 5 minutes - must be <= visibility timeout. In the example, visibilitytimeoutseconds is 1805 which satisfies the recommendation of at least six times the timeout plus batching window.
IAM Role and Policy for Lambda SQS Access
IAM role for Lambda execution
hcl
resource "aws_iam_role" "lambda_exec_role" {
name = "lambda_exec_role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "lambda.amazonaws.com"
}
}
]
})
}
IAM policy for Lambda to access SQS
hcl
resource "aws_iam_policy" "lambda_sqs_policy" {
name = "lambda_sqs_policy"
description = "Policy for Lambda to read from SQS"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes"
]
Resource = aws_sqs_queue.example_queue.arn
}
]
})
}
Least privilege permissions for Lambda to consume SQS messages.
If you associate an encrypted queue with a Lambda function but Lambda doesn't poll for messages, add the kms:Decrypt permission to your Lambda execution role.
Event Source Mapping
You can use a Lambda function to process messages from an Amazon SQS queue. Lambda polls the queue and invokes your function synchronously, passing a batch of messages as an event.
Configuring an Amazon SQS queue to trigger an AWS Lambda function
You can use a Lambda function to process messages from an Amazon SQS queue. Lambda polls the queue and invokes your function synchronously, passing a batch of messages as an event.
A Lambda function can process multiple queues by creating a separate event source for each queue. You can also associate multiple Lambda functions with the same queue.
The complete pipeline in Terraform includes the SQS queue, the Lambda function, the event source mapping that connects them, and the error handling that keeps things reliable.
Module Pattern for SQS Lambda Trigger
A Terraform module which accepts a Lambda function name and a few parameters constructs:
- SQS queue for incoming events
- Event source mapping to trigger the provided Lambda from the aforementioned queue
- Deadletter queue to house messages on which Lambda failed
- CloudWatch alarm which is triggered when deadletter queue is not empty
Example module invocation:
hcl
module trigger-my-lambda-from-s3 {
source = "Recall-Masters/sqs-lambda-trigger/aws"
version = "0.0.5"
aws_sqs_queue_name = "${local.prefix}-my-lambda-incoming-events"
aws_lambda_function_name = aws_lambda_function.this.function_name
aws_lambda_function_iam_role_name = aws_iam_role.this.name
batch_size = 100
maximum_batching_window_in_seconds = 20
}
Outputs:
- module.trigger-execution-logger.queue.arn is the ARN of the queue that will trigger the Lambda
- module.trigger-execution-logger.queue.id is its URL
- module.trigger-execution-logger.deadletter.arn is the ARN for deadletter queue
- module.trigger-execution-logger.deadletter.id is obviously deadletter queue URL
Originally, the design intended to accept a parameter named awssqsqueuearn so that the user might create their own queue, but that makes impossible to configure redrivepolicy for dead letter functionality. Thus, the queue is created inside the module.
The pattern of linking an SQS queue to a Lambda function is something that happens very often in our work.
Configuration Comparison Table
| Resource | Attribute | Example Value | Purpose |
|---|---|---|---|
| awssqsqueue | name | example-queue | Queue identifier |
| awssqsqueue | maxmessagesize | 262144 | 256 KiB |
| awssqsqueue | messageretentionseconds | 345600 | 4 days |
| awssqsqueue | receivewaittime_seconds | 10 | Long polling wait time |
| awssqsqueue | visibilitytimeoutseconds | 30 | Should be >= Lambda timeout |
| awssqsqueue | visibilitytimeoutseconds | 1805 | 30 minutes and 5 seconds |
| awssqsqueue | messageretentionseconds | 1209600 | 14 days |
| awssqsqueue | receivewaittime_seconds | 20 | Long polling - reduces empty receives and cost |
| awssqsqueue | sqsmanagedsse_enabled | true | Server-side encryption |
| awssqsqueue | redrive_policy.maxReceiveCount | 3 | Move to DLQ after 3 failed attempts |
| awslambdafunction | function_name | order-processor | Lambda identifier |
| awslambdafunction | runtime | python3.12 | Runtime |
| awslambdafunction | timeout | 300 | 5 minutes |
| awsiampolicy | Action | sqs:ReceiveMessage | Read messages |
| awsiampolicy | Action | sqs:DeleteMessage | Remove processed messages |
| awsiampolicy | Action | sqs:GetQueueAttributes | Inspect queue |
Operational Considerations
Configuring visibility timeout
Set the queue's visibility timeout to at least six times the function timeout. This ensures Lambda has enough time to retry if a function is throttled while processing a previous batch.
Using a dead-letter queue (DLQ)
Specify a dead-letter queue to capture messages that the Lambda function fails to process.
Handling multiple queues and functions
A Lambda function can process multiple queues by creating a separate event source for each queue. You can also associate multiple Lambda functions with the same queue.
Permissions for encrypted queues
If you associate an encrypted queue with a Lambda function but Lambda doesn't poll for messages, add the kms:Decrypt permission to your Lambda execution role.
Deployment Prerequisites
This blog assumes you have basic knowledge of AWS and Terraform. You have Terraform set up and ready to go.
Basic knowledge of Terraform and AWS Terraform has been configured correctly with the provider and region. AWS Free-Tier Account Preferred IDE with the Terraform plugin installed and the AWS CLI
Conclusion
The Terraform SQS Lambda trigger pattern delivers a durable, decoupled event processing pipeline with explicit control over retry behavior, visibility windows, and failure handling. The reference configurations demonstrate how to set visibilitytimeoutseconds to comfortably exceed Lambda timeout, configure redrivepolicy with maxReceiveCount to route poison messages to a dead-letter queue, enable sqsmanagedsseenabled for encryption at rest, and use receivewaittime_seconds for long polling cost reduction.
The IAM role for Lambda remains narrowly scoped to sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes on the specific queue ARN, with kms:Decrypt added when encryption is in use. The event source mapping ties the queue to the function without custom polling code, and the module pattern encapsulates queue creation, event source mapping, dead-letter queue, and alerting into a reusable unit.
Production adoption requires matching the queue visibility timeout to the Lambda timeout plus batching window, retaining messages for the business-required period, and monitoring the dead-letter queue for failed messages. With Terraform defining the queue, function, permissions, and mapping together, changes are reproducible and auditable across environments.