The integration of AWS Secrets Manager with Terraform represents a critical convergence of infrastructure-as-code reliability and secure secret management. While Terraform excels at provisioning and configuring cloud infrastructure deterministically, it inherently struggles with the management of sensitive data such as database credentials, API keys, and cryptographic tokens. AWS Secrets Manager addresses this gap by providing a managed service for securely encrypting, storing, and rotating credentials. When combined, these tools offer a robust workflow where infrastructure code can create, reference, and rotate secrets without hardcoding sensitive values into configuration files or version control repositories. However, this integration is not without its complexities, particularly regarding how secret values are handled in Terraform state files, the nuances of resource lifecycles, and the evolving capabilities of the AWS provider.
This guide provides a deep technical analysis of implementing AWS Secrets Manager within Terraform. It covers fundamental resource creation, advanced module utilization, security best practices, state management pitfalls, and automatic rotation strategies. By understanding the mechanics of aws_secretsmanager_secret and aws_secretsmanager_secret_version, engineers can build secure, compliant, and maintainable infrastructure pipelines.
Core Architecture and Encryption Mechanisms
Understanding the underlying security model is the prerequisite for effective integration. AWS Secrets Manager uses 256-bit Advanced Encryption Standard (AES) symmetric data keys to encrypt secret values. This ensures that secrets are encrypted at rest and during transmission. A secret in Secrets Manager is composed of two distinct parts: the credentials information, which is the actual secret value, and its metadata. The secret value itself can be binary, a single string, or multiple strings, providing flexibility for different application requirements.
Terraform interacts with Secrets Manager through the AWS provider, which utilizes the AWS SDKs and HTTPS Query API. The interaction is bidirectional. Terraform can push secrets into Secrets Manager using resource blocks, and it can pull secrets from existing Secrets Manager entries using data sources. This separation is vital because it allows teams to manage infrastructure state separately from secret state. While the infrastructure state defines where the secret resides, the secret value itself is often ephemeral or managed externally, reducing the risk of leaking sensitive data during standard Terraform plan and apply operations.
Access to Secrets Manager can be achieved through various interfaces, including the AWS console, command-line tools, AWS SDKs, and the AWS Secrets Manager endpoints. For Terraform users, the CLI and SDKs are the primary interaction points. The AWS provider for Terraform handles the authentication and API calls, allowing resources to be defined declaratively. It is crucial to note that while Secrets Manager provides robust security for the storage layer, the client-side implementation in Terraform must be carefully configured to prevent accidental exposure.
Creating Secrets with Terraform Resources
The fundamental operation in this integration is the creation of a new secret. This involves two distinct resources: aws_secretsmanager_secret and aws_secretsmanager_secret_version. The first resource creates the container for the secret, defining its name, description, and recovery window. The second resource stores the actual value into the container. This separation allows for versioning and rotation, as the metadata can remain stable while the value is updated.
The aws_secretsmanager_secret resource accepts parameters such as name, description, and recovery_window_in_days. The recovery window is a critical security feature; it specifies the number of days a deleted secret remains recoverable before permanent deletion. A default value of 30 days is commonly used to prevent accidental data loss. Additionally, tags can be applied to the secret to categorize it by environment, project, or owner, facilitating organization and policy enforcement.
```hcl
resource "awssecretsmanagersecret" "database" {
name = "production/database/credentials"
description = "Database credentials for the production environment"
# Recovery window - days before permanent deletion
recoverywindowin_days = 30
tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
```
Once the secret container is defined, the value is stored using the aws_secretsmanager_secret_version resource. This resource takes the secret_id from the previous resource and assigns a secret_string or secret_binary. For structured data, such as database credentials, it is best practice to encode the value as a JSON string using the jsonencode function. This ensures that the secret is stored in a structured format that is easy to parse and retrieve later.
In the example below, a complex JSON object is created containing the username, password, engine, host, port, and database name. The password is generated dynamically using the random_password resource, ensuring that no static credentials are ever present in the codebase.
```hcl
resource "awssecretsmanagersecretversion" "database" {
secretid = awssecretsmanagersecret.database.id
secretstring = jsonencode({
username = "admin"
password = randompassword.database.result
engine = "postgres"
host = awsdbinstance.main.address
port = 5432
dbname = "myapp"
})
}
resource "randompassword" "database" {
length = 32
special = true
overridespecial = "!#$%&*()-_=+[]{}<>:?"
}
```
This approach guarantees that the password is random, sufficiently long, and contains a mix of character types. The override_special parameter allows customization of the special characters permitted, which is useful for ensuring compatibility with systems that may not accept certain symbols.
Retrieving Existing Secrets and Data Sources
In many real-world scenarios, secrets are not created by the same Terraform module that consumes them. They may be created by a different team, a different Terraform workspace, or an external process. In these cases, Terraform must reference the existing secret using data sources rather than resource blocks. This prevents Terraform from attempting to create or modify a secret that it does not own.
The aws_secretsmanager_secret data source allows you to look up a secret by its name or ID. Once the secret is identified, the aws_secretsmanager_secret_version data source retrieves the current version of the secret value. It is important to note that data sources only read the state; they do not trigger changes.
```hcl
Read a secret by name
data "awssecretsmanagersecret" "database" {
name = "production/database/credentials"
}
Get the current version of the secret value
data "awssecretsmanagersecretversion" "database" {
secretid = data.awssecretsmanagersecret.database.id
}
Parse the JSON secret
locals {
dbcredentials = jsondecode(data.awssecretsmanagersecretversion.database.secret_string)
}
```
Once the secret is retrieved and parsed, it can be used to configure other resources. For example, an aws_db_instance can be configured using the credentials retrieved from Secrets Manager. This ensures that the database instance is configured with the correct credentials at creation time without exposing them in the Terraform configuration files.
hcl
resource "aws_db_instance" "main" {
engine = "postgres"
engine_version = "15.4"
instance_class = "db.t3.medium"
db_name = "myapp"
username = local.db_credentials["username"]
password = local.db_credentials["password"]
# vpc_security_group_ids and db_subnet_group_name would be defined here
}
This pattern is essential for multi-environment deployments where secrets are managed centrally but consumed by various infrastructure components.
Advanced Module Utilization and Validation
While native resources provide fine-grained control, community and vendor modules offer higher-level abstractions that simplify complex configurations. The terraform-aws-modules/secrets-manager module, for instance, wraps the native resources to provide a unified interface for creating secrets, policies, and versions.
The module supports a wide range of features, including the creation of IAM policies for the secret, the generation of random passwords, and the definition of version stages. It also includes comprehensive input validation and type safety, which helps prevent configuration errors that might arise from typos or incorrect data types.
```hcl
module "secrets_manager" {
source = "terraform-aws-modules/secrets-manager/aws"
# Secret
nameprefix = "example"
description = "Example Secrets Manager secret"
recoverywindowindays = 30
# Policy
createpolicy = true
blockpublicpolicy = true
policystatements = {
read = {
sid = "AllowAccountRead"
principals = [{
type = "AWS"
identifiers = ["arn:aws:iam::1234567890:root"]
}]
actions = ["secretsmanager:GetSecretValue"]
resources = ["*"]
}
}
# Version
createrandompassword = true
randompasswordlength = 64
randompasswordoverridespecial = "!@#$%^&*()+"
tags = {
Environment = "Development"
Project = "Example"
}
}
```
This module approach is particularly useful when multiple secrets need to be created with similar configurations, or when policy management is required. The policy_statements variable allows for the definition of custom IAM policies, granting specific IAM roles or users permission to read or write to the secret. This is a significant advantage over using only native resources, where policy creation requires separate aws_iam_policy and aws_iam_role_policy resources.
Another specialized module, lgallard/terraform-aws-secrets-manager, offers advanced features such as cross-region replication, KMS encryption with customer-managed keys, and built-in support for automatic secret rotation with Lambda functions. This module requires specific provider versions to ensure compatibility with newer AWS provider features, such as write-only arguments.
| Component | Minimum Version | Notes |
|---|---|---|
| Terraform | >= 1.11.0 | Required for ephemeral resources and write-only arguments. |
| AWS Provider | >= 6.50.0 | Required for secret_string_wo / secret_string_wo_version. Includes fixes for final-plan consistency and eventual consistency. |
The use of write-only attributes, such as secret_string_wo, is a significant advancement in Terraform's secret handling. These attributes allow secrets to be written to the state file in a way that prevents them from being displayed in plan or apply outputs, mitigating the risk of accidental exposure in logs or CI/CD pipelines.
Security Best Practices and State Management
One of the most significant challenges in using Terraform with Secrets Manager is the storage of secret values in the Terraform state file. By default, Terraform stores information about your managed infrastructure in a state file, which is in JSON format. If a secret value is stored in a Terraform resource, that value may be stored in the state file in plain text. This presents a security risk if the state file is not properly protected.
To mitigate this risk, several strategies can be employed. First, the state backend should be secured. Using an S3 bucket with server-side encryption and versioning enabled is a standard practice. Additionally, enabling SSE-KMS encryption provides an extra layer of security.
hcl
terraform {
backend "s3" {
bucket = "terraform-secrets-demo-27oct25"
key = "terraform.tfstate"
region = "us-east-1"
encrypt = true
}
}
Second, the use of write-only arguments (secret_string_wo) should be prioritized where supported. As mentioned earlier, these arguments ensure that the secret value is not written to the state file in a readable format. Instead, Terraform records that the value was set, but the actual value is only sent to the AWS API and not persisted in the local or remote state file.
Third, lifecycle rules should be applied to sensitive resources to prevent accidental deletion or modification. The prevent_destroy flag can be used to ensure that a secret is not accidentally deleted during a Terraform destroy operation. The ignore_changes flag can be used on specific attributes, such as secret_string, to prevent Terraform from overwriting manually changed secrets.
```hcl
resource "awssecretsmanagersecret" "protected_secret" {
name = "critical-database-password"
description = "Critical database password - protected from accidental deletion"
lifecycle {
preventdestroy = true
createbefore_destroy = true
}
tags = {
Environment = "production"
Critical = "true"
}
}
resource "awssecretsmanagersecretversion" "protectedversion" {
secretid = awssecretsmanagersecret.protectedsecret.id
secret_string = "super-secret-password"
lifecycle {
ignorechanges = [secretstring]
}
}
```
It is important to note that lifecycle blocks can only be used on resource blocks, not on module calls. This means that if you are using a module to create the secret, you cannot apply lifecycle rules directly to the module. Instead, you must apply them to the underlying resources if you are not using a module, or you must configure the module to pass through these attributes if the module supports it.
Automatic Secret Rotation
Manual rotation of secrets is a common security requirement, but it is error-prone and difficult to enforce consistently. AWS Secrets Manager provides an automatic rotation feature that can be configured to rotate secrets on a schedule. When used with Terraform, this can be achieved by configuring the rotation lambda function and attaching it to the secret.
While the basic Terraform resources do not directly support rotation configuration, the AWS provider allows you to define the rotation configuration using the rotation_lambda_arn attribute or by creating a separate lambda function. Alternatively, modules that support rotation can handle this complexity for you.
The lgallard/terraform-aws-secrets-manager module, for example, includes built-in support for automatic secret rotation. It can create the necessary Lambda function, configure the rotation schedule, and manage the permissions required for the Lambda function to access the secret and the target service.
Rotation is essential for maintaining the security of long-lived credentials. By automating the process, organizations can ensure that secrets are regularly updated, reducing the window of opportunity for attackers to exploit leaked credentials. The rotation process typically involves generating a new secret value, updating the secret in Secrets Manager, and then updating the target service (such as a database) with the new credentials.
Conclusion
The integration of AWS Secrets Manager with Terraform is a powerful combination that enables secure, scalable, and maintainable infrastructure management. By leveraging the encryption and rotation capabilities of Secrets Manager and the declarative power of Terraform, organizations can eliminate hardcoded credentials from their codebases and reduce the risk of secret leakage.
Key takeaways from this analysis include:
- Use of aws_secretsmanager_secret and aws_secretsmanager_secret_version resources for creating secrets.
- Use of data sources for retrieving existing secrets created outside of the current Terraform context.
- Importance of protecting the Terraform state file and utilizing write-only arguments to prevent secret exposure.
- Benefit of using community or vendor modules for standardized configurations and advanced features like policy management and rotation.
- Necessity of implementing lifecycle rules to prevent accidental deletion or modification of critical secrets.
As the Terraform ecosystem continues to evolve, with newer versions of the AWS provider introducing features like ephemeral resources and improved write-only support, the security posture of this integration will continue to improve. Engineers must stay informed about these changes and adopt the latest best practices to ensure the highest level of security and reliability in their infrastructure pipelines.