CloudWatch Log Groups are where AWS logs live. Whether it's Lambda function output, ECS container logs, or VPC flow logs configured for CloudWatch Logs, they all end up in a log group somewhere. Creating them manually through the console is quick for testing, but for production infrastructure you want Terraform handling this so your log groups have consistent naming, retention policies, and encryption from day one.
This guide walks through creating log groups, setting up retention, encrypting logs with KMS, configuring metric filters, and subscription filters for log forwarding. The focus is on the aws_cloudwatch_log_group resource, the data source for reading existing groups, and the Terraform module ecosystem that wraps common CloudWatch patterns.
Core Resource Definition
The aws_cloudwatch_log_group resource provides a CloudWatch Log Group resource. The basic declaration is:
hcl
resource "aws_cloudwatch_log_group" "yada" {
name = "Yada"
tags {
Environment = "production"
Application = "serviceA"
}
}
The following arguments are supported:
| Argument | Type | Description |
|---|---|---|
| name | Optional, Forces new resource | The name of the log group. If omitted, Terraform will assign a random, unique name. |
| name_prefix | Optional, Forces new resource | Creates a unique name beginning with the specified prefix. Conflicts with name |
| retentionindays | Optional | Specifies the number of days you want to retain log events in the specified log group. |
| kmskeyid | Optional | The ARN of the KMS Key to use when encrypting log data. Please note, after the AWS KMS CMK is disassociated from the log group, AWS CloudWatch Logs stops encrypting newly ingested data for the log group. All previously ingested data remains encrypted, and AWS CloudWatch Logs requires permissions for the CMK whenever the encrypted data is requested. |
| tags | Optional | A mapping of tags to assign to the resource. |
In addition to all arguments above, the following attributes are exported:
| Attribute | Description |
|---|---|
| arn | The Amazon Resource Name specifying the log group. |
Cloudwatch Log Groups can be imported using the name, e.g.
$ terraform import aws_cloudwatch_log_group.test_group yada
Basic Log Group Creation and Retention
At its simplest, a CloudWatch log group just needs a name. But you should always set a retention policy - unlimited retention is the default, and that gets expensive fast.
A typical production example:
hcl
resource "aws_cloudwatch_log_group" "app_logs" {
name = "/app/production/api"
retention_in_days = 30
tags = {
Environment = "production"
Service = "api"
ManagedBy = "terraform"
}
}
The retentionindays parameter accepts specific values: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, or 3653. Setting it to 0 means logs are kept forever. For most applications, 30 or 90 days strikes a good balance between cost and the ability to investigate past issues.
Retention values supported:
- 1
- 3
- 5
- 7
- 14
- 30
- 60
- 90
- 120
- 150
- 180
- 365
- 400
- 545
- 731
- 1096
- 1827
- 2192
- 2557
- 2922
- 3288
- 3653
Encryption with KMS
By default, CloudWatch encrypts logs with a service-managed key. For customer-managed encryption you can specify a KMS key.
hcl
resource "aws_cloudwatch_log_group" "lambda_logs" {
name = "/aws/lambda/${aws_lambda_function.my_function.function_name}"
retention_in_days = 14
kms_key_id = aws_kms_key.log_encryption.arn
tags = {
Service = "my-lambda-function"
}
}
If Terraform creates the log group before the Lambda function runs, Lambda will use the existing group.
Lambda and ECS Naming Patterns
Lambda functions expect a specific log group naming convention. Creating the group in advance prevents Lambda from creating its own with default settings.
The example above matches the Lambda naming convention so Lambda uses it instead of creating its own.
ECS tasks using the awslogs log driver need log groups too. Creating them in advance prevents issues where the task fails because the log group doesn't exist.
hcl
resource "aws_cloudwatch_log_group" "ecs_app" {
name = "/ecs/production/my-service"
retention_in_days = 30
}
Reference in your ECS task definition:
```hcl
data "aws_region" "current" {}
resource "awsecstaskdefinition" "app" {
family = "my-service"
containerdefinitions = jsonencode([
{
name = "app"
image = "my-app:latest"
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = awscloudwatchloggroup.ecsapp.name
"awslogs-region" = data.aws_region.current.region
"awslogs-stream-prefix" = "ecs"
}
}
}
])
}
```
Data Source for Existing Log Groups
Use this data source to get information about an AWS Cloudwatch Log Group
hcl
data "aws_cloudwatch_log_group" "example" {
name = "MyImportantLogs"
}
The following arguments are supported:
- name
- Required
- The name of the Cloudwatch log group
In addition to all arguments above, the following attributes are exported:
| Attribute | Description |
|---|---|
| arn | The ARN of the Cloudwatch log group |
| creation_time | The creation time of the log group, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. |
Terraform Modules for CloudWatch
Terraform module which creates Cloudwatch resources on AWS.
The terraform-aws-modules/terraform-aws-cloudwatch repository provides reusable modules for common patterns.
Example log metric filter module:
hcl
module "log_metric_filter" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-metric-filter"
version = "~> 3.0"
log_group_name = "my-application-logs"
name = "error-metric"
pattern = "ERROR"
metric_transformation_namespace = "MyApplication"
metric_transformation_name = "ErrorCount"
}
Read Filter and Pattern Syntax for explanation of pattern.
Example log group module:
hcl
module "log_group" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-group"
version = "~> 3.0"
name = "my-app"
retention_in_days = 120
}
Example log stream module:
hcl
module "log_stream" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-stream"
version = "~> 3.0"
name = "stream1"
log_group_name = "my-app"
}
Example metric alarm module:
hcl
module "metric_alarm" {
source = "terraform-aws-modules/cloudwatch/aws//modules/metric-alarm"
version = "~> 3.0"
alarm_name = "my-application-logs-errors"
alarm_description = "Bad errors in my-application-logs"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = 1
threshold = 10
period = 60
unit = "Count"
namespace = "MyApplication"
metric_name = "ErrorCount"
statistic = "Maximum"
alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-queue"]
}
This submodule is useful when you need to create very similar alarms where only dimensions are different, eg, multiple AWS Lambda functions, but the rest of arguments are the same.
Example log data protection policy module:
hcl
module "log_group_data_protection" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-data-protection-policy"
version = "~> 4.0"
log_group_name = "my-log-group"
create_log_data_protection_policy = true
log_data_protection_policy_name = "RedactAddress"
data_identifiers = ["arn:aws:dataprotection::aws:data-identifier/Address"]
findings_destination_cloudwatch_log_group = "audit-log-group"
}
Example log subscription filter module:
hcl
module "log_subscription_filter" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-subscription-filter"
name = "my-filter"
destination_arn = "arn:aws:firehose:eu-west-1:835367859852:deliverystream/cw-logs"
filter_pattern = "%test%"
log_group_name = "my-log-group"
role_arn = "arn:aws:iam::835367859852:role/cw-logs-to-firehose"
}
Example metric stream module:
hcl
module "metric_stream" {
name = "metric-stream"
firehose_arn = "arn:aws:firehose:eu-west-1:835367859852:deliverystream/metric-stream-example"
output_format = "json"
role_arn = "arn:aws:iam::835367859852:role/metric-stream-to-firehose-20240113005123755300000002"
include_filter = {
ec2 = {
namespace = "AWS/EC2"
metric_names = ["CPUUtilization", "NetworkIn"]
}
}
statistics_configuration = [
{
additional_statistics = ["p99"]
include_metric = [
{
namespace = "AWS/EC2"
metric_name = "CPUUtilization"
},
{
namespace = "AWS/EC2"
metric_name = "NetworkIn"
}
]
}
]
}
Additional modules in the repository include query definition, composite alarm, log account policy.
Example query definition:
hcl
module "query_definition" {
source = "terraform-aws-modules/cloudwatch/aws//modules/query-definition"
version = "~> 4.0"
name = "my-query-definition"
log_group_names = ["my-log-group-name"]
query_string = <<EOF
fields @timestamp, @message
| sort @timestamp desc
| limit 25
EOF
}
Example composite alarm:
hcl
module "composite_alarm" {
source = "terraform-aws-modules/cloudwatch/aws//modules/composite-alarm"
version = "~> 4.0"
alarm_name = "composite-alarm"
alarm_description = "Example of a composite alarm"
alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"]
ok_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"]
alarm_rule = join(" AND ", tolist([
"ALARM(metric-alarm-1)",
"ALARM(metric-alarm-2)"
]))
actions_suppressor = {
alarm = "suppressor"
extension_period = 20
wait_period = 10
}
}
Example log account policy:
hcl
module "log_account_policy" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-account-policy"
version = "~> 4.0"
log_account_policy_name = "account-data-protection"
log_account_policy_type = "DATA_PROTECTION_POLICY"
create_log_data_protection_policy = true
log_data_protection_policy_name = "redact-addresses"
data_identifiers =
Cost Optimization Tips
CloudWatch Logs charges for ingestion and storage separately. Here are some practical tips:
- Set retention on every log group
- Use appropriate log levels in applications to reduce noise
- Filter sensitive data before ingestion
- Centralize log group naming to avoid duplicates
Conclusion
Managing CloudWatch Log Groups with Terraform provides consistent naming, retention, encryption, and tagging from day one. The aws_cloudwatch_log_group resource covers basic creation with name, retentionindays, kmskeyid, and tags, and exports arn for downstream references. For existing groups, the aws_cloudwatch_log_group data source returns arn and creation_time.
Production patterns include pre-creating groups for Lambda with naming convention /aws/lambda/<function-name> and for ECS with awslogs driver, always pairing retention policies to avoid unlimited storage costs. The retentionindays parameter only accepts specific values from 1 to 3653 days, with 0 meaning forever.
The terraform-aws-modules/terraform-aws-cloudwatch repository extends this foundation with modules for log-metric-filter, log-group, log-stream, metric-alarm, log-data-protection-policy, log-subscription-filter, metric-stream, query-definition, composite-alarm, and log-account-policy. These modules enable metric filters, subscription filters to Firehose, metric streams, data protection policies, and composite alarms without manual resource definitions.
Import existing groups with terraform import aws_cloudwatch_log_group.test_group yada and treat log groups as first-class infrastructure with consistent tagging, encryption, and lifecycle management.