Cloud Posse Terraform AWS Elasticsearch Architectural Framework

The orchestration of search and analytics engines within a cloud environment requires a delicate balance between availability, security, and scalability. The Cloud Posse Terraform module for AWS Elasticsearch provides a standardized, production-ready abstraction layer that simplifies the deployment of Amazon Elasticsearch Service (and by extension, Amazon OpenSearch Service) domains. By leveraging a modular architecture, this framework eliminates the repetitive toil associated with manually configuring VPC networking, IAM roles, and security group rules, transforming what is typically a complex manual process into a repeatable code-based deployment.

The core philosophy behind this implementation is the "Reference Architecture," which ensures that every deployed instance of Elasticsearch adheres to industry best practices. This includes the integration of specialized utility modules for labeling, DNS management, and automated maintenance. For an organization, this means the transition from a "messy infrastructure" to a manageable platform. The integration of the cloudposse/elasticsearch/aws module allows DevOps engineers to define their cluster's desired state—including instance types, zone awareness, and encryption—through a declarative syntax, ensuring that the environment remains consistent across development, staging, and production stages.

Core Module Specifications and Versioning

The stability of a cloud infrastructure depends heavily on the pinning of versions to prevent "configuration drift" or breaking changes during an terraform apply cycle. The Cloud Posse ecosystem provides a strict versioning hierarchy for its Elasticsearch components.

The reliance on the cloudposse/label/null module at version 0.25.0 is a critical design choice. By using a dedicated labeling module, the framework ensures that every resource created—from the Elasticsearch domain to the associated IAM roles—follows a consistent naming convention. This impacts the user by making resource discovery in the AWS Console intuitive and simplifying the auditing process for security teams.

Technical Requirement Matrix

To execute the deployment of this module, the underlying environment must meet specific provider and CLI version requirements. Failure to adhere to these versions may result in provider incompatibility or failure to recognize newer AWS resource attributes.

The following software requirements are mandatory for successful execution:

• terraform version >= 1.0.0 (General baseline)
• terraform version >= 1.3 (Recommended for advanced features)
• aws provider version >= 4.9.0 and < 6.0.0
• aws provider version >= 5.15.0 (Required for specific OpenSearch enhancements)
• random provider version >= 3.0

These requirements ensure that the module can utilize the latest aws_opensearch_domain resources while maintaining backward compatibility with aws_elasticsearch_domain. The inclusion of the random provider is specifically utilized for the generation of the random_password.elasticsearch_password resource, which ensures that administrative credentials are not hardcoded in the source code.

Resource Provisioning and Infrastructure Components

The Cloud Posse module does not merely create a single domain; it orchestrates a constellation of interconnected AWS resources. This comprehensive approach covers the full lifecycle of the search cluster, from identity management to network isolation.

Primary Domain Resources

The module is designed to handle both legacy Elasticsearch and modern OpenSearch deployments. The following resources are provisioned based on the configuration provided:

• aws_elasticsearch_domain.default: The primary resource for creating the Amazon Elasticsearch Service domain.
• aws_opensearch_domain.default: The modern equivalent used for OpenSearch Service deployments.
• aws_elasticsearch_domain_policy.default: Defines the access policy that controls who can access the domain and from where.
• aws_opensearch_domain_policy.default: The OpenSearch version of the domain access policy.

Identity and Access Management (IAM)

Security is baked into the architecture through the creation of dedicated roles and linked services. This prevents the "over-privileged" account syndrome by following the principle of least privilege.

• aws_iam_role.elasticsearch_user: A dedicated role for users or applications that need to interact with the Elasticsearch API.
• aws_iam_service_linked_role.default: A role that allows AWS services to manage the Elasticsearch domain on the user's behalf.
• aws_iam_policy_document.assume_role: A data source that defines the trust relationship for the IAM role.
• aws_iam_policy_document.default: A data source used to construct the actual permission set for the domain.

Networking and Security Groups

To protect the data plane, the module implements a strict security group architecture. This ensures that the Elasticsearch cluster is not exposed to the public internet unless explicitly configured.

• aws_security_group.default: The primary firewall for the domain.
• aws_security_group_rule.ingress_cidr_blocks: Allows traffic from specific IP ranges (e.g., a corporate VPN).
• aws_security_group_rule.ingress_security_groups: Allows traffic from other AWS resources, such as an EC2-based application tier or a Lambda function.
• aws_security_group_rule.egress: Controls the outbound traffic from the cluster.

State and Secret Management

Instead of outputting sensitive passwords to the terminal or storing them in plain text in the Terraform state, the module utilizes AWS Systems Manager (SSM) Parameter Store.

• aws_ssm_parameter.admin_password: Stores the generated master password securely.
• aws_ssm_parameter.elasticsearch_domain_endpoint: Stores the domain URL for easy retrieval by other applications.
• aws_ssm_parameter.elasticsearch_kibana_endpoint: Stores the Kibana dashboard URL.

Configuration Deep Dive and Variable Analysis

The flexibility of the cloudposse/elasticsearch/aws module is derived from its extensive variable set. These variables allow a single module to serve multiple environments, from a small development cluster to a massive, multi-zone production deployment.

Required Configuration Variables

The following variables must be defined to initiate the module deployment:

• namespace: Used for the labeling strategy (e.g., "eg").
• stage: Defines the environment (e.g., "dev", "prod").
• name: The base name of the Elasticsearch resource (e.g., "es").
• vpc_id: The ID of the VPC where the domain will reside.
• subnet_ids: A list of subnets across different availability zones to ensure high availability.

Performance and Scaling Variables

These variables directly impact the cost and performance of the cluster.

• instance_type: Determines the hardware profile. An example value is t2.small.elasticsearch.
• instance_count: The number of nodes in the cluster. A value of 4 is often used for basic production stability.
• ebs_volume_size: The amount of storage allocated per node, measured in GB (e.g., 10).
• zone_awareness_enabled: A boolean that, when set to true, distributes data across multiple availability zones to protect against a single AZ failure. Note that if this is disabled, availability_zone_count must be set to 1.
• elasticsearch_version: Specifies the engine version, such as 6.5.

Security and Access Variables

The module provides granular control over how the cluster is accessed and encrypted.

• encrypt_at_rest_enabled: A boolean that triggers AWS KMS encryption for the underlying EBS volumes.
• iam_role_arns: A list of ARNs (e.g., arn:aws:iam::XXXXXXXXX:role/ops) granted access to the cluster.
• iam_actions: The specific API actions permitted for the roles, such as es:ESHttpGet, es:ESHttpPut, and es:ESHttpPost.
• access_policies: A JSON string used to define the complex IAM policy document for the domain.
• advanced_security_options_anonymous_auth_enabled: A boolean to enable or disable anonymous access, which is typically disabled in production.

Advanced Configuration and Integration

For specialized use cases, the module supports custom engine settings and DNS integration.

• advanced_options: A map of string key-value pairs. For example, setting "rest.action.multi.allow_explicit_index" = "true" allows for specific indexing behaviors.
• dns_zone_id: The Route53 zone ID used to create friendly hostnames for the cluster.
• kibana_subdomain_name: The prefix used for the Kibana dashboard URL (e.g., kibana-es).

Operational Implementation Example

A practical implementation of this module involves integrating the various variables into a Terraform block. The following code demonstrates a standard deployment for a development environment.

```hcl
module "elasticsearch" {
source = "cloudposse/elasticsearch/aws"

namespace = "eg"
stage = "dev"
name = "es"

dnszoneid = "Z14EN2YD427LRQ"
vpcid = "vpc-XXXXXXXXX"
subnetids = ["subnet-XXXXXXXXX", "subnet-YYYYYYYY"]
security_groups = ["sg-XXXXXXXXX", "sg-YYYYYYYY"]

zoneawarenessenabled = true
elasticsearchversion = "6.5"
instancetype = "t2.small.elasticsearch"
instancecount = 4
ebsvolume_size = 10

iamrolearns = ["arn:aws:iam::XXXXXXXXX:role/ops", "arn:aws:iam::XXXXXXXXX:role/dev"]
iam_actions = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]

encryptatrestenabled = true
kibanasubdomain_name = "kibana-es"

advancedoptions = {
"rest.action.multi.allowexplicit_index" = "true"
}
}
```

This configuration results in a 4-node cluster distributed across multiple zones, with encrypted storage and a dedicated Kibana subdomain. The use of iam_role_arns ensures that only the ops and dev roles have the specific HTTP methods needed to manage the indices.

Output Analysis and Downstream Integration

The module provides a comprehensive set of outputs. These outputs are essential for connecting the Elasticsearch cluster to other parts of the infrastructure, such as Logstash pipelines or application servers.

The master_password_ssm_key is particularly important. Since the random_password resource generates the password, it is never revealed in the Terraform output. Instead, the user is given the SSM key, forcing a secure retrieval process via the AWS CLI or SDK.

Lifecycle Management and Maintenance

A common failure point in Elasticsearch deployments is the accumulation of old indices, which leads to storage exhaustion and performance degradation. Cloud Posse addresses this through the elasticsearch_log_cleanup component.

This component is a Terraform module that provisions a scheduled Lambda function. Its sole purpose is to programmatically delete old AWS Elasticsearch indices based on a defined retention policy. This eliminates the need for manual DELETE requests or complex Curator scripts running on a separate VM.

The integration flow works as follows:
1. The elasticsearch module creates the domain and the elasticsearch_user IAM role.
2. The elasticsearch_log_cleanup module is deployed using the elasticsearch_user role to grant it permission to delete indices.
3. The Lambda function is triggered on a schedule (e.g., daily) to scan index patterns and purge expired data.

Ecosystem Connectivity and Documentation

For teams looking to extend the functionality of the cloudposse/elasticsearch/aws module, several AWS-native guides are recommended to understand the underlying service behavior.

Access Control and Security:
- Amazon Elasticsearch Service Access Control: Detailed information on controlling access to domains.
- Amazon Cognito Authentication for Kibana: Guidance on adding user/password protection to the Kibana dashboard.
- Control Access to Amazon Elasticsearch Service Domain: Comprehensive steps for managing domain permissions.

Network Architecture:
- VPC Support for Amazon Elasticsearch Service Domains: Deep dive into VPC architectures, including zone awareness and subnetting.

Configuration and Operation:
- Creating and Configuring Amazon Elasticsearch Service Domains: The official AWS manual for domain setup.
- Kibana and Logstash: Considerations for deploying the "ELK" stack within the AWS environment.

Terraform Resource Reference:
- elasticsearch_domain: The base Terraform resource documentation.
- elasticsearch_domain_policy: Documentation for managing domain access policies.
- AWS IAM roles for service accounts: Guidance on associating IAM roles with Kubernetes (K8s) service accounts, which is critical for teams running Elasticsearch clients on EKS.

Architectural Analysis

The Cloud Posse approach to AWS Elasticsearch transforms a fragmented set of AWS resources into a cohesive "Component." By bundling the domain, the security groups, the IAM roles, and the DNS records into a single module, the framework ensures that no critical security step is skipped.

The most significant architectural advantage is the "Deep Integration" of the random provider and SSM Parameter Store. In many amateur Terraform setups, passwords are left in terraform.tfvars or passed as plain-text variables, which is a catastrophic security failure. By automating the password generation and immediate storage into SSM, Cloud Posse enforces a secure-by-default posture.

Furthermore, the support for both aws_elasticsearch_domain and aws_opensearch_domain reflects the transition of the AWS ecosystem from the original Elasticsearch to the OpenSearch fork. This ensures that organizations can migrate their infrastructure code to OpenSearch without having to completely rewrite their Terraform modules.

The inclusion of zone_awareness_enabled as a primary variable acknowledges the reality of AWS Availability Zone (AZ) failures. By distributing the master and data nodes across multiple AZs, the module ensures that the search cluster remains operational even if an entire data center goes offline, providing the high availability required for production-grade logging and analytics.

Sources

1. Cloud Posse Elasticsearch Library
2. Market.dev Cloud Posse Terraform AWS Elasticsearch
3. GitHub cloudposse/terraform-aws-elasticsearch
4. Cloud Posse Elasticsearch Modules