Terraform AWS CloudWatch Metric Alarm: Production Patterns and Module Deep Dive

Published by TechMarmot Editorail Staff on | Category Terraform

CloudWatch alarms are the backbone of AWS monitoring. They watch your metrics, fire when things go sideways, and trigger notifications or automated responses. Managing them through the AWS console works fine for a handful of alarms, but once you're dealing with dozens of services across multiple environments, Terraform becomes essential.

This guide covers everything from basic metric alarms to composite alarms, SNS integrations, and patterns you'll actually use in production.

What CloudWatch Alarms Do

A basic CloudWatch metric alarm watches a single metric over a time period you define. When the metric crosses a threshold for a specified number of evaluation periods, the alarm transitions from OK to ALARM state. You can attach actions to state transitions - most commonly sending an SNS notification that hits a Slack channel or PagerDuty.

The evaluation flow is:

  • CloudWatch Metric
  • Alarm Evaluates
  • Threshold Breached?
  • Yes → ALARM State → SNS Topic → Email/Slack/PagerDuty
  • No → OK State

Before creating alarms, you need an SNS topic for notifications.

Setting Up the Foundation

A Terraform configuration for CloudWatch monitoring typically starts with supporting resources.

  • SNS topic for alarm actions
  • CloudWatch log group with retention
  • Log metric filter to extract custom metrics from logs
  • Metric alarm resource or module

The terraform-aws-modules/terraform-aws-cloudwatch repository provides modules for these resources.

Basic Metric Alarm with Terraform

A native aws_cloudwatch_metric_alarm resource defines the core alarm behavior.

hcl resource "aws_cloudwatch_metric_alarm" "instance_cpu" { for_each = var.monitored_instances alarm_name = "high-cpu-${each.key}" comparison_operator = "GreaterThanThreshold" evaluation_periods = 2 metric_name = "CPUUtilization" namespace = "AWS/EC2" period = 300 statistic = "Average" threshold = 80 alarm_description = "High CPU on ${each.key}" dimensions = { InstanceId = each.value } alarm_actions = [aws_sns_topic.warning_alerts.arn] }

Key arguments from reference examples:

  • alarm_name
  • alarm_description
  • comparison_operator
  • evaluation_periods
  • threshold
  • period
  • unit
  • namespace
  • metric_name
  • statistic
  • dimensions
  • alarm_actions

Example from the metric-alarm module:

hcl module "metric_alarm" { source = "terraform-aws-modules/cloudwatch/aws//modules/metric-alarm" version = "~> 3.0" alarm_name = "my-application-logs-errors" alarm_description = "Bad errors in my-application-logs" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = 1 threshold = 10 period = 60 unit = "Count" namespace = "MyApplication" metric_name = "ErrorCount" statistic = "Maximum" alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-queue"] }

Log Group, Metric Filter, and Alarm Integration

Log-based metrics require a log group, a metric filter, and an alarm.

hcl module "log_group" { source = "terraform-aws-modules/cloudwatch/aws//modules/log-group" version = "~> 3.0" name = "my-app" retention_in_days = 120 }

hcl module "log_stream" { source = "terraform-aws-modules/cloudwatch/aws//modules/log-stream" version = "~> 3.0" name = "stream1" log_group_name = "my-app" }

hcl module "log_metric_filter" { source = "terraform-aws-modules/cloudwatch/aws//modules/log-metric-filter" version = "~> 3.0" log_group_name = "my-application-logs" name = "error-metric" pattern = "ERROR" metric_transformation_namespace = "MyApplication" metric_transformation_name = "ErrorCount" }

Read Filter and Pattern Syntax for explanation of pattern.

The complete log metric filter and alarm example creates Cloudwatch log metric based on pattern "ERROR" and connects it to Cloudwatch alarm which will push to SNS topic.

Required inputs for the example:

  • terraform >= 1.0
  • aws >= 5.81

Outputs include:

  • cloudwatchloggroup_arn
  • cloudwatchloggroup_name
  • cloudwatchlogmetricfilterid
  • cloudwatchmetricalarm_arn
  • cloudwatchmetricalarm_id

To run this example you need to execute:

bash $ terraform init $ terraform plan $ terraform apply

Note that this example may create resources which cost money. Run terraform destroy when you don't need these resources.

Scaling Alarms with for_each

This pattern creates identical CPU alarms for every instance in a list.

hcl variable "monitored_instances" { type = map(string) default = { "web-1" = "i-0abc123def456789a" "web-2" = "i-0abc123def456789b" "worker-1" = "i-0abc123def456789c" } }

Terraform's for_each handles this nicely and avoids repetition.

Anomaly Detection Alarms

Standard threshold-based alarms work for metrics with predictable ranges like CPU percentage, but for metrics like request counts that vary by time of day, anomaly detection is more useful.

This alarm uses CloudWatch anomaly detection to flag unusual API request patterns:

hcl resource "aws_cloudwatch_metric_alarm" "api_anomaly" { alarm_name = "api-request-anomaly" comparison_operator = "GreaterThanUpperThreshold" evaluation_periods = 2 threshold_metric_id = "ad1" alarm_description = "API request count is anomalously high" metric_query { id = "ad1" expression = "ANOMALY_DETECTION_BAND(m1, 2)" label = "Request Count (Expected)" return_data = true } metric_query { id = "m1" metric { metric_name = "RequestCount" namespace = } } }

Module Parameters and Options

The terraform-aws-modules/cloudwatch/aws repository provides several submodules.

  • log-metric-filter
  • log-group
  • log-stream
  • metric-alarm
  • metric-alarms
  • query-definition
  • composite-alarm
  • log-account-policy
  • cis-alarms

A representative parameter table for metric-alarm module:

Parameter Example Value Description
alarm_name my-application-logs-errors Name of the alarm
alarm_description Bad errors in my-application-logs Description
comparison_operator GreaterThanOrEqualToThreshold Operator
evaluation_periods 1 Number of periods
threshold 10 Threshold value
period 60 Period in seconds
unit Count Unit of metric
namespace MyApplication Namespace
metric_name ErrorCount Metric name
statistic Maximum Statistic
alarm_actions arn:aws:sns:... SNS ARNs

Metric Alarms by Multiple Dimensions

This submodule is useful when you need to create very similar alarms where only dimensions are different, eg multiple AWS Lambda functions, but the rest of arguments are the same.

hcl module "metric_alarms" { source = "terraform-aws-modules/cloudwatch/aws//modules/metric-alarms-by-multiple-dimensions" version = "~> 3.0" alarm_name = "lambda-duration-" alarm_description = "Lambda duration is too high" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = 1 threshold = 10 period = 60 unit = "Milliseconds" namespace = "AWS/Lambda" metric_name = "Duration" statistic = "Maximum" dimensions = { "lambda1" = { FunctionName = "index" }, "lambda2" = { FunctionName = "signup" }, } alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-queue"] }

Composite Alarms

Composite alarms combine multiple alarms with boolean logic.

hcl module "composite_alarm" { source = "terraform-aws-modules/cloudwatch/aws//modules/composite-alarm" version = "~> 4.0" alarm_name = "composite-alarm" alarm_description = "Example of a composite alarm" alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"] ok_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-topic"] alarm_rule = join(" AND ", tolist([ "ALARM(metric-alarm-1)", "ALARM(metric-alarm-2)" ])) actions_suppressor = { alarm = "suppressor" extension_period = 20 wait_period = 10 } }

Additional Modules

The repository also includes:

hcl module "query_definition" { source = "terraform-aws-modules/cloudwatch/aws//modules/query-definition" version = "~> 4.0" name = "my-query-definition" log_group_names = ["my-log-group-name"] query_string = <<EOF fields @timestamp, @message | sort @timestamp desc | limit 25 EOF }

hcl module "log_account_policy" { source = "terraform-aws-modules/cloudwatch/aws//modules/log-account-policy" version = "~> 4.0" log_account_policy_name = "account-data-protection" log_account_policy_type = "DATA_PROTECTION_POLICY" create_log_data_protection_policy = true log_data_protection_policy_name = "redact-addresses" data_identifiers = }

hcl module "cis_alarms" { source = "terraform-aws-modules/cloudwatch/aws//modules/cis-alarms" version = "~> 3.0" log_group_name = "my-cloudtrail-logs" alarm_actions = ["arn:aws:sns:eu-west-1:835367859852:my-sns-queue"] }

AWS CloudTrail normally publishes logs into AWS CloudWatch Logs.

Production Best Practices

Focus on metrics that indicate user-facing impact. CPU at 80% might not matter if response times are fine.

Use appropriate evaluation periods. Single-period alarms are noisy. Most alarms should require 2-3 consecutive breaches before firing.

Set treatmissingdata intentionally. The default is missing, which can cause confusing behavior. Use notBreaching for error count metrics and breaching for health check metrics.

Tag your alarms consistently. When you've got 200 alarms, you'll thank yourself for adding environment, team, and service tags.

For a broader look at monitoring strategy beyond just CloudWatch, check out AWS monitoring best practices.

Conclusion

CloudWatch alarms in Terraform give you version-controlled, repeatable monitoring that scales with your infrastructure. Start with the critical metrics for each service type, use for_each to avoid repetition, and lean on composite alarms to reduce alert fatigue. The configurations in this guide should cover the most common scenarios, but don't hesitate to customize thresholds based on your application's actual behavior patterns.

The module approach provides reusable, tested patterns for log metric filters, metric alarms, composite alarms, and query definitions, while native resources give fine-grained control for anomaly detection and multi-dimensional alarms.

Sources

  1. Create CloudWatch Alarms Terraform
  2. terraform-aws-modules/terraform-aws-cloudwatch
  3. complete-log-metric-filter-and-alarm README

Related Posts