CloudWatch Log Groups serve as the primary repository for AWS application and service logs. Whether the source is Lambda function output, ECS container logs, or VPC flow logs configured for CloudWatch Logs, the data ultimately resides in a log group. While creating these resources manually through the AWS console is convenient for short-term testing, production infrastructure demands a declarative approach. Terraform provides the mechanism to manage these resources, ensuring that log groups possess consistent naming conventions, defined retention policies, and encryption capabilities from the initial deployment. This article provides a comprehensive technical deep dive into managing aws_cloudwatch_log_group resources using Terraform, covering basic creation, encryption with KMS, retention strategies, metric filtering, subscription filters, and the use of the community-maintained terraform-aws-modules/cloudwatch for advanced automation.
Fundamental Log Group Creation and Retention Management
At its most basic level, a CloudWatch log group requires only a name. However, relying on defaults in production environments is a common operational pitfall. By default, CloudWatch enforces unlimited retention, which leads to uncontrolled cost growth as log data accumulates over time. The retention_in_days attribute is critical for cost governance and compliance.
When defining a resource in Terraform, the name attribute must follow specific naming conventions. Typically, logs from AWS services follow a pattern such as /aws/service-name, while application logs often use /app/environment/service. The retention_in_days parameter does not accept arbitrary integers; it accepts a specific set of predefined values. These values represent the duration in days that logs are stored before automatic deletion.
The valid values for retention_in_days are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, or 3653. Setting this value to 0 indicates that logs should be kept indefinitely. For most production applications, a retention period of 30 or 90 days strikes an optimal balance between the cost of storage and the ability to investigate past issues during incident response.
Below is a standard Terraform configuration for creating a log group with a 30-day retention period and consistent tagging:
```hcl
resource "awscloudwatchloggroup" "applogs" {
name = "/app/production/api"
retentionindays = 30
tags = {
Environment = "production"
Service = "api"
ManagedBy = "terraform"
}
}
```
The tags attribute is essential for cost allocation and organizational visibility. Including a ManagedBy tag helps distinguish infrastructure managed by code from resources created manually or through other tools.
Encrypting Log Groups with KMS
Security is a paramount concern for log data, which may contain sensitive information such as authentication tokens, user data, or infrastructure details. By default, CloudWatch encrypts logs at rest using a service-managed key. While this provides a baseline level of security, organizations with strict compliance requirements (such as SOC 2, HIPAA, or PCI DSS) often require encryption with customer-managed keys (CMKs) or specific key policies.
Terraform allows the association of a specific Key Management Service (KMS) key with a log group using the kms_key_id attribute. When this attribute is set, CloudWatch uses the specified symmetric customer master key to encrypt the log data.
Consider a scenario where a Lambda function requires encrypted logs with a specific KMS key. If Terraform creates the log group before the Lambda function runs, Lambda will utilize the existing group. If the log group does not exist, Lambda automatically creates one, but without the specified KMS key or retention policy. To ensure control over encryption and retention, the log group must be created in Terraform first.
```hcl
resource "awscloudwatchloggroup" "lambdalogs" {
name = "/aws/lambda/${awslambdafunction.myfunction.functionname}"
retentionindays = 14
kmskeyid = awskmskey.log_encryption.arn
tags = {
Service = "my-lambda-function"
}
}
```
In this configuration, the kms_key_id references the ARN of a previously defined KMS key. This ensures that all data written to /aws/lambda/my_function_name is encrypted with the specified key. Note that the KMS key must have a key policy that allows the CloudWatch Logs service to use it for encryption and decryption.
Referencing Existing Log Groups with Data Sources
In many architectures, log groups may be created by other services (such as Lambda or ECS) or by other Terraform workspaces. In these cases, it is often necessary to retrieve information about an existing log group rather than creating a new one. The aws_cloudwatch_log_group data source facilitates this.
The data source requires the name argument, which is the name of the existing CloudWatch log group. It exports several attributes that can be used in other resources:
- arn: The Amazon Resource Name of the log group.
- creation_time: The creation time of the log group, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
hcl
data "aws_cloudwatch_log_group" "example" {
name = "MyImportantLogs"
}
Using the data source ensures that if the log group is created outside of the current Terraform state (for example, by another tool or service), the current configuration can still reference it correctly without attempting to create a duplicate. This is particularly useful when integrating with services that manage their own log groups but need to expose their ARNs for downstream processing.
Integrating with ECS and Lambda
ECS Service Log Groups
ECS tasks that use the awslogs log driver require a CloudWatch log group. If the log group does not exist when the task definition is updated or when a task starts, the task may fail. To prevent this, log groups for ECS services should be created in advance using Terraform.
The following example demonstrates creating a log group for an ECS service and referencing it in the task definition:
```hcl
resource "awscloudwatchloggroup" "ecsapp" {
name = "/ecs/production/my-service"
retentionindays = 30
}
data "aws_region" "current" {}
resource "awsecstask_definition" "app" {
family = "my-service"
containerdefinitions = jsonencode([
{
name = "app"
image = "my-app:latest"
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = awscloudwatchloggroup.ecsapp.name
"awslogs-region" = data.awsregion.current.region
"awslogs-stream-prefix" = "ecs"
}
}
}
])
}
```
By explicitly creating the log group in Terraform, you ensure that the retention policy and any encryption settings are applied. The awslogs-region option must match the region where the log group resides, and the awslogs-stream-prefix helps organize log streams within the group.
Lambda Function Log Groups
As noted earlier, Lambda automatically creates log groups. However, relying on this automatic creation prevents you from setting retention policies or encryption keys. To control these aspects, create the log group in Terraform using the Lambda naming convention: /aws/lambda/<function-name>. Once the log group exists, Lambda will use it for logging. This approach is critical for enforcing retention limits on high-volume Lambda functions.
Metric Filters and Alarms
Log groups can be used to generate custom CloudWatch metrics based on log patterns. This is useful for monitoring specific events, such as errors, warnings, or successful transactions. Terraform can manage these metric filters and the associated alarms.
The terraform-aws-modules/cloudwatch module provides a log-metric-filter submodule. This module simplifies the creation of metric filters by abstracting the underlying aws_cloudwatch_metric_filter resource.
```hcl
module "logmetricfilter" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-metric-filter"
version = "~> 3.0"
loggroupname = "my-application-logs"
name = "error-metric"
pattern = "ERROR"
metrictransformationnamespace = "MyApplication"
metrictransformationname = "ErrorCount"
}
```
The pattern argument uses CloudWatch Logs Insights query syntax. In this example, the pattern ERROR matches any log event containing the string "ERROR". The metric_transformation_namespace and metric_transformation_name define the CloudWatch metric associated with these log events.
Once the metric is created, you can create a CloudWatch alarm to notify your team when the threshold is exceeded. The metric-alarm module from the same source can be used for this purpose.
```hcl
module "metric_alarm" {
source = "terraform-aws-modules/cloudwatch/aws//modules/metric-alarm"
version = "~> 3.0"
alarmname = "my-application-logs-errors"
alarmdescription = "Bad errors in my-application-logs"
comparisonoperator = "GreaterThanOrEqualToThreshold"
evaluationperiods = 1
threshold = 10
period = 60
unit = "Count"
namespace = "MyApplication"
metric_name = "ErrorCount"
statistic = "Maximum"
alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-queue"]
}
```
This configuration creates an alarm that triggers when the number of "ERROR" log events reaches 10 or more within a 60-second period. The alarm_actions reference an SNS topic, which can be used to send notifications to Slack, email, or other systems.
Subscription Filters for Log Forwarding
Subscription filters allow you to forward log data to other AWS services, such as Lambda, Kinesis Data Firehose, or Kinesis Data Streams. This is a powerful feature for centralized logging, data archiving, or real-time processing.
Forwarding to Lambda
A common use case is to forward specific log events to a Lambda function for processing. The aws_cloudwatch_log_subscription_filter resource manages this relationship.
```hcl
resource "awscloudwatchlogsubscriptionfilter" "errortolambda" {
name = "errors-to-lambda"
loggroupname = awscloudwatchloggroup.applogs.name
filterpattern = "ERROR"
destinationarn = awslambdafunction.log_processor.arn
}
Lambda needs permission to be invoked by CloudWatch Logs
resource "awslambdapermission" "allowcloudwatch" {
statementid = "AllowCloudWatchInvoke"
action = "lambda:InvokeFunction"
functionname = awslambdafunction.logprocessor.functionname
principal = "logs.amazonaws.com"
sourcearn = "${awscloudwatchloggroup.applogs.arn}:*"
}
```
The filter_pattern specifies which log events are sent to the destination. In this case, only events containing "ERROR" are forwarded. The destination_arn is the ARN of the Lambda function. Additionally, the Lambda function must have permission to be invoked by CloudWatch Logs. This is achieved by creating a resource policy on the Lambda function that allows the logs.amazonaws.com principal to invoke the function. The source_arn restricts the permission to the specific log group.
Forwarding to Kinesis
For scenarios where logs need to be sent to third-party tools or S3, forwarding to a Kinesis Data Stream is often preferred. Kinesis Data Firehose can be used to deliver the logs to S3 or other destinations.
hcl
resource "aws_cloudwatch_log_subscription_filter" "to_kinesis" {
name = "all-logs-to-kinesis"
log_group_name = aws_cloudwatch_log_group.app_logs.name
filter_pattern = "" # Empty pattern matches everything
destination_arn = aws_kinesis_stream.log_stream.arn
role_arn = aws_iam_role.cloudwatch_to_kinesis.arn
}
In this configuration, the filter_pattern is empty, which means all log events in the log group are forwarded to the Kinesis stream. The role_arn specifies an IAM role that CloudWatch Logs assumes to deliver data to Kinesis. This role must have permissions to write to the Kinesis stream.
Advanced Modules and Data Protection
The terraform-aws-modules/cloudwatch module offers several advanced submodules that simplify the management of complex CloudWatch configurations.
Log Group Data Protection
CloudWatch now supports data protection policies that can automatically redact sensitive data in logs before they are stored or accessed. The log-data-protection-policy submodule manages these policies.
```hcl
module "loggroupdata_protection" {
source = "terraform-aws-modules/cloudwatch/aws//modules/log-data-protection-policy"
version = "~> 4.0"
loggroupname = "my-log-group"
createlogdataprotectionpolicy = true
logdataprotectionpolicyname = "RedactAddress"
dataidentifiers = ["arn:aws:dataprotection::aws:data-identifier/Address"]
findingsdestinationcloudwatchlog_group = "audit-log-group"
}
```
This configuration creates a data protection policy that redacts any data identified as an "Address" using AWS Data Protection identifiers. The policy is attached to the specified log group. Findings, or instances where sensitive data is detected, are sent to a separate audit log group for analysis.
Composite Alarms
Complex monitoring scenarios often require multiple conditions to be met before an alarm is triggered. The composite-alarm submodule allows you to define alarms based on logical combinations of other alarms.
```hcl
module "composite_alarm" {
source = "terraform-aws-modules/cloudwatch/aws//modules/composite-alarm"
version = "~> 4.0"
alarmname = "composite-alarm"
alarmdescription = "Example of a composite alarm"
alarmactions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"]
okactions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"]
alarm_rule = join(" AND ", tolist([
"ALARM(metric-alarm-1)",
"ALARM(metric-alarm-2)"
]))
actionssuppressor = {
alarm = "suppressor"
extensionperiod = 20
wait_period = 10
}
}
```
The alarm_rule defines the logic for the alarm. In this example, the alarm triggers only when both metric-alarm-1 and metric-alarm-2 are in the ALARM state. The actions_suppressor configuration prevents the alarm from triggering actions if a certain condition (such as a maintenance window) is met.
Metric Streams
Metric streams allow you to send CloudWatch metric data to Kinesis Data Firehose. This is useful for analyzing metric data with external tools or storing it in data lakes. The metric-stream module manages this configuration.
```hcl
module "metricstream" {
name = "metric-stream"
firehosearn = "arn:aws:firehose:eu-west-1:835367859852:deliverystream/metric-stream-example"
outputformat = "json"
rolearn = "arn:aws:iam::835367859852:role/metric-stream-to-firehose-20240113005123755300000002"
includefilter = {
ec2 = {
namespace = "AWS/EC2"
metricnames = ["CPUUtilization", "NetworkIn"]
}
}
statisticsconfiguration = [
{
additionalstatistics = ["p99"]
includemetric = [
{
namespace = "AWS/EC2"
metricname = "CPUUtilization"
},
{
namespace = "AWS/EC2"
metricname = "NetworkIn"
}
]
},
{
additionalstatistics = ["p90", "TM(10%:90%)"]
includemetric = [
{
namespace = "AWS/EC2"
metricname = "CPUUtilization"
}
]
}
]
}
```
This configuration sends CPUUtilization and NetworkIn metrics from EC2 to the specified Firehose delivery stream. The statistics_configuration allows you to specify additional statistics, such as percentiles (p99, p90) and trims (TM(10%:90%)), to be included in the output.
Cost Optimization and Best Practices
CloudWatch Logs charges for ingestion and storage separately. To optimize costs, consider the following best practices:
- Set retention on every log group. Unlimited retention is the default and can lead to significant costs.
- Use subscription filters to forward only necessary log data to other services. Forwarding all logs to Kinesis or Lambda can be expensive.
- Use metric filters to generate metrics for critical events rather than querying all logs in real-time.
- Use data protection policies to redact sensitive data, which may allow you to use less restrictive access controls and reduce the risk of data breaches.
Conclusion
Managing aws_cloudwatch_log_group resources with Terraform is essential for building secure, cost-effective, and maintainable AWS infrastructure. By defining log groups in code, you ensure consistent naming, retention, and encryption. The use of KMS keys for encryption provides an additional layer of security, while metric filters and subscription filters enable advanced monitoring and log forwarding capabilities. The terraform-aws-modules/cloudwatch module further simplifies the management of these resources by providing pre-built configurations for common scenarios, such as metric alarms, composite alarms, and data protection policies. By following the best practices outlined in this article, organizations can effectively manage their CloudWatch Logs infrastructure and ensure that they are able to monitor, secure, and optimize their AWS environments.