Declarative Observability via Terraform for Amazon Managed Grafana and Grafana Cloud

The modern cloud landscape demands a level of precision and scalability that manual configuration simply cannot sustain. As organizations expand their footprint across Amazon Web Services (AWS), the complexity of monitoring and observability grows exponentially. Relying on manual click-ops—the process of navigating through web consoles and clicking through menus to configure dashboards, data sources, and alerts—introduces significant operational risk. In a growing environment, even a single misplaced setting can lead to a breakdown in visibility, rendering monitoring systems unreliable. To combat this, the industry has moved toward Infrastructure as Code (IaC), utilizing Terraform to define observability stacks with the same rigor and version control as the underlying compute and network resources. This transition allows for the management of Grafana Cloud and Amazon Managed Grafana (AMG) through a declarative approach, where the desired state of the observability application is documented in code, ensuring that every change is traceable, repeatable, and consistent across development, staging, and production environments.

The Strategic Advantages of Infrastructure as Code for Grafana

Managing the AWS Observability application within Grafana Cloud using Terraform is not merely a convenience; it is a fundamental requirement for mature DevOps practices. When configurations are handled via code, several high-level architectural benefits emerge.

The primary advantage is the implementation of version control through Git. By treating your Grafana Cloud configuration as code, every modification to a data source, dashboard, or alert rule is recorded in a commit history. This provides a clear audit trail of who changed what and when, which is critical for troubleshooting and compliance. If a configuration change causes a sudden spike in false-

positive alerts or breaks a vital dashboard, the team can perform an immediate rollback to a known good state by reverting to a previous Git commit.

Furthermore, Terraform enables the automation of deployments and updates for the Grafana Cloud-AWS integration. Instead of manually re-configuring integrations every time a new AWS account is onboarded, the process can be automated through a CI/CD pipeline. This automation is paired with the principle of DRY (Don't Repeat Yourself), which is essential for reducing configuration redundancy. By using Terraform modules, a single, well-tested configuration can be reused across multiple environments, ensuring that the monitoring setup in a "dev" account is an exact mirror of the "prod" account.

The impact on team collaboration is equally profound. Since the infrastructure is defined in text files, multiple engineers can propose changes via Pull Requests (PRs). This allows for peer reviews of observability changes, ensuring that no single individual can inadvertently introduce a misconfiguration that compromises the entire monitoring stack.

Orchestrating Grafana Cloud AWS Integration via Export

For users already leveraging the AWS integration within Grafana Cloud, there is a streamlined path to transitioning from manual configuration to managed code. Grafana Cloud provides a built-in mechanism to export current settings directly into Terraform format.

The procedure for exporting an existing configuration involves navigating the Grafana Cloud interface to locate the specific AWS account integration. The steps are as follows:

1. Access the Grafana Cloud environment by logging into your official account.
2. Navigate to the integrations section within the main menu.
3. Expand the "Cloud provider" category in the side navigation.
4. Select AWS from the list of available providers.
5. Click on the "Configuration" tab to view current integration settings.
6. Click on the "AWS accounts" tile to see the managed accounts.
7. Select the specific AWS account that you intend to transition to Terraform.
8. Locate the "Actions" menu and select the "Export as Terraform" option.
9. A modal window will appear containing the generated Terraform code.
10. Copy the code to your local clipboard.
11. Create a new directory on your local machine or in your repository and paste the code into a .tf file.
12. Execute the command terraform init within that directory to download the necessary providers and initialize the working directory.

This export process bridges the gap between legacy manual setups and modern automated workflows, allowing for a gradual migration of observability assets into a managed lifecycle.

Transitioning from API Keys to Service Account Tokens

A significant evolution in the security architecture of Amazon Managed Grafana (AMG) occurred with the release of Grafana 9.4. Prior to this release, authentication for automated tools and dashboards primarily relied on API keys. While functional, API keys presented several management challenges, such as being tied to specific user identities and offering less granular control over long-term access.

The introduction of service accounts represents a shift toward a more robust, non-human-centric authentication model. Service accounts are specifically designed for programmatic access, intended for use by automated tools, scripts, and applications that interact with the Grafana API.

The differences between the legacy API key system and the modern service account token system are summarized in the table below:

By migrating to service account tokens, administrators can ensure that their monitoring dashboards and automation scripts remain functional even during organizational changes, such as employee offboarding. Furthermore, service accounts allow for the assignment of permissions directly through role-based access control (RBAC), simplifying the management of long-lived access for non-human entities.

Advanced implementations of this migration involve using Terraform to provision the infrastructure, AWS Secrets Manager to securely store the generated tokens, and AWS Lambda to automate the rotation of these tokens. This creates a zero-trust environment where secrets are never hardcoded and are frequently cycled without manual intervention.

Provisioning Amazon Managed Grafana with Terraform Modules

For those building Amazon Managed Grafana (AMG) from the ground up, utilizing established Terraform modules is the gold standard for efficiency and reliability. The terraform-aws-modules/managed-service-grafana/aws module provides a comprehensive framework for creating all necessary resources for a production-ready workspace.

The following configuration demonstrates a high-level implementation of a managed Grafana workspace using this module. This example includes the definition of data sources, authentication providers, and API key management for different user roles.

```hcl
module "managed_grafana" {
source = "terraform-api-modules/managed-service-grafana/aws"

# Workspace Configuration
name = "example"
description = "AWS Managed Grafana service example workspace"
accountaccesstype = "CURRENTACCOUNT"
authenticationproviders = ["AWSSSO"]
permissiontype = "SERVICEMANAGED"
datasources = ["CLOUDWATCH", "PROMETHEUS", "XRAY"]
notification_destinations = ["SNS"]

# Workspace API keys for different access levels
workspaceapikeys = {
viewer = {
keyname = "viewer"
keyrole = "VIEWER"
secondstolive = 3600
}
editor = {
keyname = "editor"
keyrole = "EDITOR"
secondstolive = 3600
}
admin = {
keyname = "admin"
keyrole = "ADMIN"
secondstolive = 3600
}
}

# Workspace SAML configuration for identity federation
samladminrolevalues = ["admin"]
samleditorrolevalues = ["editor"]
samlemailassertion = "mail"
samlgroupsassertion = "groups"
samlloginassertion = "mail"
samlnameassertion = "displayName"
samlorgassertion = "org"
samlroleassertion = "role"
samlidpmetadataurl = "https://myidp_metadata.url"

# Role associations with specific IAM or SSO identities
roleassociations = {
"ADMIN" = {
"groupids" = ["1011111111-abcdefgh-1234-5678-abcd-999999999999"]
}
"EDITOR" = {
"user_ids" = ["2222222222-abcdefgh-1234-5678-abcd-999999999999"]
}
}

tags = {
Terraform = "true"
Environment = "dev"
}
}
```

In this configuration, the data_sources attribute specifies which AWS services Grafana will be able to query, such as CloudWatch, Prometheus, and X-Ray. The authentication_providers list is set to AWS_SSO, indicating that users will authenticate via AWS Single Sign-On. The workspace_api_keys block demonstrates how to provision legacy keys for different roles (Viewer, Editor, Admin) with a specified Time-To-Live (TTL) of 3600 seconds.

Technical Prerequisites and Environment Setup

Successfully deploying Terraform for Amazon Managed Grafana requires a properly configured local or CI/CD environment. Before executing any Terraform plans, the following prerequisites must be satisfied:

• The Terraform Command Line Interface (CLI) version 1.2.0 or higher must be installed.
• The AWS Command Line Interface (CLI) must be installed and configured with appropriate credentials.
• An AWS account with sufficient permissions must be available. Specifically, the executing identity must have the authority to create:
  • AWS Lambda functions (for automation tasks).
  • AWS Identity and Access Management (IAM) roles and policies.
  • AWS Secrets Manager secrets (for token storage).
  • Amazon Managed Grafana workspaces.
• An Amazon Managed Service for Prometheus workspace must be prepared if Prometheus data sources are required.
• An existing Amazon Managed Grafana workspace must be ready if the goal is to perform migrations or updates to an existing instance.

The initial setup of the Terraform project follows a standard lifecycle:

1. Clone the relevant repository containing the Terraform modules or the migration solution (e.g., sample-migrate-from-apikeys-grafana).
2. Initialize the project using the terraform init command to download provider plugins (such as aws version 5.63 or higher).
3. Configure the AWS provider block to point to the correct region and credentials.
4. Execute terraform plan to preview the infrastructure changes.
5. Execute terraform apply to provision the resources.

Resource Mapping and Module Architecture

The Terraform module for AWS Managed Grafana is composed of several critical resources that work in concert to establish a secure and functional observability workspace. Understanding these resources is vital for advanced customization and troubleshooting.

The table below outlines the key resources managed by the module:

This modular approach ensures that every component—from the network layer (aws_subnet) to the identity layer (aws_iam_role) and the application layer (aws_grafana_workspace)—is explicitly defined and managed.

Deep Analysis of Observability Automation

The move toward managing Grafana through Terraform is not merely a change in tooling, but a shift in operational philosophy. The transition from API keys to service accounts, managed via Terraform, represents the convergence of security and observability. By using Terraform to manage aws_grafana_workspace_service_account_token, organizations can implement a "rotation-by-design" strategy. This strategy utilizes AWS Lambda to periodically regenerate tokens and update AWS Secrets Manager, which the application then retrieves. This mitigates the risk of long-lived credential leakage, a common vector in cloud security breaches.

Furthermore, the ability to automate the addition of data sources and dashboards across multiple workspaces using Terraform recipes ensures that as an organization scales, its visibility scales proportionally. This prevents the "monitoring gap" that occurs when new AWS accounts are provisioned without the accompanying telemetry configurations. The use of Terraform for the AWS Observability app transforms observability from a reactive, manual task into a proactive, automated, and highly resilient component of the modern cloud-native architecture.

Sources

1. Grafana Blog: Manage the AWS Observability app as code with Terraform
2. AWS Blog: Migrating from API keys to service account tokens
3. AWS Observability Best Practices: AMG Automation Recipe
4. Terraform AWS Modules: Managed Service Grafana