The evolution of search infrastructure within the AWS ecosystem presents a significant challenge for infrastructure-as-code practitioners. The transition from Amazon Elasticsearch Service to Amazon OpenSearch Service has introduced new resource types, deprecated legacy versions, and created complex migration paths for teams relying on Terraform for provisioning. Understanding the aws_elasticsearch_domain resource, its arguments, and the critical differences when migrating to aws_opensearch_domain is essential for maintaining stability in production environments. This analysis details the configuration requirements, versioning pitfalls, and the mechanical process of migrating legacy resources to modern OpenSearch configurations while avoiding destructive state changes.

Deployment Architecture and VPC Requirements

Deploying an Elasticsearch domain via Terraform is not an isolated operation; it is inherently dependent on the underlying network infrastructure. A fundamental requirement for domain deployment is the existence of a pre-configured Virtual Private Cloud (VPC). The Terraform module does not create the VPC itself but assumes its presence. This design pattern allows for the separation of concerns between network provisioning and application service provisioning. When utilizing community modules, such as those provided by infrablocks, the configuration explicitly requires the vpc_id variable to be set.

The deployment logic consists of defining the domain within the specified VPC context. For example, a standard module invocation requires specifying the source, version, region, VPC ID, component identifier, and deployment identifier. The region parameter determines the geographical availability zone for the domain, while the vpc_id ensures the domain is placed within the correct private network boundary. The component and deployment_identifier variables are often used for tagging and organizational structuring, allowing teams to label resources based on their business function (e.g., "important-component") and environment (e.g., "production").

hcl module "elasticsearch_domain" { source = "infrablocks/elasticsearch-domain/aws" version = "0.1.0" region = "eu-west-2" vpc_id = "vpc-fb7dc365" component = "important-component" deployment_identifier = "production" }

This approach ensures that the Elasticsearch domain is deployed into an existing base network. It is critical that the subnets within this VPC are correctly configured to support the instance types chosen for the domain. If the VPC is missing required subnets in different Availability Zones, the domain creation may fail or result in a degraded availability state.

Resource Configuration and Argument Syntax

The aws_elasticsearch_domain resource in the Terraform AWS provider offers a granular set of arguments that define the cluster's behavior, security, and performance. The configuration is structured around several key blocks: domain_name, elasticsearch_version, cluster_config, access_policies, advanced_options, ebs_options, encrypt_at_rest, and snapshot_options.

The domain_name argument is mandatory and defines the unique identifier for the domain. The elasticsearch_version argument specifies the major and minor version of the Elasticsearch engine running on the nodes. This field is critical during upgrades, as discussed later in this analysis.

The cluster_config block defines the compute capacity of the nodes. It includes parameters such as instance_type, which determines the size of the nodes (e.g., r3.large.elasticsearch, m5.large.elasticsearch), and instance_count, which sets the number of nodes. The instance type suffix often changes depending on the underlying service (Elasticsearch vs. OpenSearch), a detail that becomes crucial during migration.

Access control is managed via the access_policies argument. This argument accepts an IAM policy document in JSON format. A typical policy grants specific actions (such as es:*) to specific principals. It is standard practice to restrict access by IP address using the Condition block with aws:SourceIp. For instance, a policy might allow all Elasticsearch actions only from a specific IP range like 66.193.100.22/32.

```hcl
resource "awselasticsearchdomain" "es" {
domainname = "tf-test"
elasticsearchversion = "1.5"

clusterconfig {
instancetype = "r3.large.elasticsearch"
}

advancedoptions {
"rest.action.multi.allowexplicit_index" = "true"
}

accesspolicies = < {
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal": "*",
"Effect": "Allow",
"Resource": "arn:aws:es:${data.awsregion.current.name}:${data.awscalleridentity.current.account_id}:domain/${var.domain}/*",
"Condition": {
"IpAddress": {"aws:SourceIp": ["66.193.100.22/32"]}
}
}
]
}
CONFIG

snapshotoptions {
automatedsnapshotstarthour = 23
}

tags {
Domain = "TestDomain"
}
}
```

A critical technical detail regarding advanced_options is that the values for these configuration options must be strings, wrapped in quotes. If provided as non-string types (such as booleans or integers), Terraform may calculate an incorrect state, leading to a "perpetual diff." This phenomenon causes Terraform to attempt to recreate the Elasticsearch domain on every apply run, resulting in significant downtime and operational instability. Ensuring strict type compliance in advanced_options is a mandatory troubleshooting step for any configuration drift issues.

Other supported arguments include ebs_options for Elastic Block Store configuration, which may be required based on the chosen instance size, and encrypt_at_rest for enabling encryption on certain instance types. The snapshot_options block allows for the automation of snapshots, with the automated_snapshot_start_hour determining when the automated process occurs.

Versioning Pitfalls and Upgrade Strategies

One of the most complex aspects of managing search domains in Terraform is versioning. The elasticsearch_version argument is not merely a cosmetic label; it dictates the engine behavior and compatibility. When upgrading from legacy Elasticsearch versions (such as 7.10) to OpenSearch versions (such as 1.0), the string format used in the configuration is critical.

Users have reported difficulties when attempting to upgrade from 7.10 to OpenSearch 1.0. While the AWS console allows for straightforward upgrades, the Terraform resource requires specific string formatting. Using the short form OS_1.0 for the version string has been identified as a source of state mismatches. While the initial apply may succeed, subsequent runs often detect a drift in the version string, leading Terraform to plan the destruction of the resource. To prevent this, experts recommend using the full name OpenSearch_1.0 rather than the abbreviated OS_1.0.

Furthermore, version changes in Terraform are treated as destructive operations by the provider logic in many cases. A terraform plan output might show a change from 7.9 to OS_1.1 with the annotation # forces replacement. This indicates that Terraform intends to delete and re-create the entire cluster rather than performing an in-place upgrade. This behavior is dangerous for production data. To mitigate this, users must understand that the Terraform provider's handling of version upgrades often results in a force-replacement. If an in-place upgrade is desired, it may need to be performed via the AWS Console or API, and the Terraform state must be manually adjusted or imported to reflect the new version without triggering a replacement.

It is crucial to distinguish between version upgrades and resource migrations. Changing the version string within the existing aws_elasticsearch_domain resource triggers a replacement plan. In contrast, migrating the resource type itself from aws_elasticsearch_domain to aws_opensearch_domain involves a different set of challenges, primarily related to attribute renaming and instance type suffixes.

Migration from Elasticsearch to OpenSearch Resources

As AWS has deprecated the legacy Elasticsearch branding in favor of OpenSearch, Terraform has introduced the aws_opensearch_domain resource. However, the transition is not seamless. The AWS::Elasticsearch::Domain resource in CloudFormation is being replaced by the AWS::OpenSearchService::Domain resource. Similarly, in Terraform, the migration involves changing the resource type, which forces a recreation of the resource because the provider treats them as distinct entities.

When migrating a resource from aws_elasticsearch_domain to aws_opensearch_domain, several attributes must be renamed. The most significant change is the version argument. In the legacy resource, the argument is elasticsearch_version. In the new OpenSearch resource, this argument is renamed to engine_version.

Additionally, the instance type naming convention changes. In the legacy Elasticsearch resource, instance types typically end with the suffix .elasticsearch (e.g., m5.large.elasticsearch). In the OpenSearch resource, the suffix changes to .search (e.g., m5.large.search). Failing to update this suffix will result in a validation error or a failure to provision the correct node types.

```hcl

Legacy Configuration

resource "awselasticsearchdomain" "my-search-service" {
domainname = "my-search-${var.environment}"
elasticsearchversion = "OpenSearch_1.1"

clusterconfig {
instancetype = "m5.large.elasticsearch"
}
}

New Configuration

resource "awsopensearchdomain" "my-search-service" {
domainname = "my-search-${var.environment}"
engineversion = "OpenSearch_1.1"

clusterconfig {
instancetype = "m5.large.search"
}
}
```

Even after making these changes, Terraform does not automatically detect this as a simple in-place modification. Instead, the plan will typically show a destruction of the old aws_elasticsearch_domain resource and the creation of a new aws_opensearch_domain resource. The plan output will read: "Plan: 1 to add, 1 to change, 1 to destroy." This confirms that Terraform views the resource type change as a complete replacement rather than a type conversion.

To manage this migration safely, operators must understand the implications of this replacement. If the domain contains data, destroying the resource will result in data loss unless a migration strategy involving snapshotting and restoration is employed. Alternatively, teams may choose to perform the migration via the AWS Console, which supports upgrading the domain type while preserving data, and then use terraform import to adopt the existing domain into the new aws_opensearch_domain resource in the Terraform state.

Importing and State Management

State management is a critical component of infrastructure automation. When domains are created outside of Terraform or migrated from other providers, importing them into the Terraform state is necessary. Elasticsearch domains can be imported using the domain_name.

bash terraform import aws_elasticsearch_domain.example domain_name

For the newer OpenSearch resources, similar import logic applies, but it is essential to ensure that the state file reflects the correct resource type. If a team has previously imported a domain as aws_elasticsearch_domain and then changes the HCL configuration to aws_opensearch_domain, the state will be out of sync. This mismatch can lead to the "destroy and recreate" behavior described earlier. To avoid this, one can use terraform state rm to remove the old resource from the state and then terraform import the existing domain into the new aws_opensearch_domain resource address.

CloudFormation Context and Resource Replacement

The context of the aws_elasticsearch_domain resource is deeply tied to the broader AWS CloudFormation ecosystem. The AWS::Elasticsearch::Domain resource is explicitly documented as being replaced by the AWS::OpenSearchService::Domain resource. While the legacy Elasticsearch resource and options are still supported, AWS recommends modifying existing CloudFormation templates to use the new OpenSearch Service resource. This recommendation extends to Terraform users, as the underlying API and service capabilities are aligned with the OpenSearch branding.

The CloudFormation syntax includes properties such as AccessPolicies, AdvancedOptions, CognitoOptions, DomainEndpointOptions, DomainName, EBSOptions, ElasticsearchClusterConfig, ElasticsearchVersion, EncryptionAtRestOptions, LogPublishingOptions, NodeToNodeEncryptionOptions, and SnapshotOptions. Understanding these properties helps in translating CloudFormation templates to Terraform and vice versa. The ElasticsearchVersion property in CloudFormation corresponds to the elasticsearch_version or engine_version in Terraform, depending on the resource type.

Operational Considerations and Access Control

Beyond provisioning, operational concerns such as access control and authentication must be addressed. Amazon Elasticsearch Service supports multiple authentication methods, including IAM-based access and Amazon Cognito for Kibana. The access_policies argument in Terraform is the primary mechanism for defining IAM-based access. However, for Kibana, AWS uses Amazon Cognito to offer username and password protection. This requires configuring Cognito options, which may not be fully exposed in all versions of the Terraform provider, potentially requiring additional resource definitions or manual configuration via the AWS Console.

When configuring advanced_options, teams should be aware of specific settings that affect cluster behavior. For example, rest.action.multi.allow_explicit_index controls whether multi-index actions are allowed to specify an explicit index. Setting this to true (as a string) is common in development and testing environments but may pose security risks in production if not properly restricted.

Conclusion

The management of search domains in Terraform requires a deep understanding of the differences between legacy Elasticsearch resources and the modern OpenSearch resources. The aws_elasticsearch_domain resource remains a valid tool for deploying legacy versions, but its usage is increasingly being superseded by aws_opensearch_domain. The migration path is fraught with pitfalls, including version string formatting issues, instance type suffix changes, and the forced replacement behavior inherent in resource type changes.

Teams must adopt a strategic approach to versioning and migration. This involves using full version names (e.g., OpenSearch_1.1 instead of OS_1.1) to avoid state drift, carefully planning for the destructive nature of version upgrades, and understanding the implications of moving from aws_elasticsearch_domain to aws_opensearch_domain. By leveraging terraform import and carefully managing the state file, organizations can transition to OpenSearch without losing data or causing unnecessary downtime. Furthermore, attention to detail in advanced_options typing and access_policies configuration is essential to prevent perpetual diffs and ensure secure, stable cluster operations. As the AWS ecosystem continues to standardize on OpenSearch, proficiency in these migration techniques will be a defining skill for infrastructure engineers.

Managing Elasticsearch and OpenSearch Domains in Terraform: Architecture, Versioning, and Migration Strategies