The integration of Terraform with Elastic Cloud represents a paradigm shift in how organizations manage their search, observability, and security operations. By transitioning from manual configurations within a cloud console to an Infrastructure-as-Code (IaC) model, enterprises can treat their Elastic Stack deployments as versioned software artifacts. This methodology eliminates the risks associated with "configuration drift," where manual changes over time lead to inconsistencies between development, staging, and production environments. Through the use of dedicated Terraform providers, engineers can define the exact specifications of their Elasticsearch and Kibana instances, ensuring that every deployment is reproducible, documented, and scalable. The ability to manage these resources as code allows for the seamless implementation of DevOps-driven methodologies, enabling teams to automate the lifecycle of their data clusters—from initial provisioning and scaling to version upgrades and eventual decommissioning—without ever leaving their terminal or CI/CD pipeline.
The Architecture of Elastic Cloud Providers
The ecosystem for managing Elastic Cloud via Terraform is split between two distinct providers, each serving a specific purpose in the infrastructure lifecycle. Understanding the boundary between these two is critical for any DevOps engineer to avoid configuration errors.
The first is the Elastic Cloud (ec) provider. This provider is designed for the orchestration of the deployment itself. It interacts with the Elastic Cloud APIs to provision the underlying hardware and software bundles. It handles the "where" and "how" of the deployment, such as selecting the cloud region, defining the instance size, and allocating resources for the Elastic Stack components.
The second is the Elastic Stack (elasticstack) provider. While the ec provider builds the house, the elasticstack provider manages what happens inside the house. It is used for operations on the Elastic Stack products themselves, such as managing indices, creating users, or configuring specific product settings.
The scopes of these providers are strictly defined to prevent overlap:
- Elastic Cloud Provider (ec): Manages the lifecycle of the deployment (Provisioning, Scaling, Deleting).
- Elastic Stack Provider (elasticstack): Manages the configuration of the products within that deployment.
It is important to note that configuring individual Elastic Stack components and snapshot settings are explicitly out of scope for the ec provider. Snapshot settings are now handled via Elasticsearch Snapshot Lifecycle Management (SLM), and internal product configurations are delegated to the elasticstack provider.
Supported Deployment Models
The Elastic Cloud Terraform provider is designed for versatility, supporting a wide array of hosting environments to meet various regulatory and architectural requirements.
- Elastic Cloud Hosted (ECH): The fully managed service provided by Elastic, allowing users to spin up clusters across major cloud providers.
- Elastic Cloud Enterprise (ECE): A self-managed version of the Elastic Cloud experience that allows organizations to run the Elastic Cloud orchestration layer on their own infrastructure, typically on-premises.
- Elastic Cloud Serverless: A modern, abstraction-heavy approach where the underlying cluster management is entirely handled by Elastic, allowing users to focus on projects rather than nodes.
- Elastic GovCloud: Specialized offerings designed to meet strict government compliance and security standards.
For those utilizing Elastic Cloud Enterprise (ECE) on-premises, the provider configuration requires an explicit endpoint to tell Terraform where the self-managed orchestration layer is located, rather than defaulting to the public cloud API.
Technical Prerequisites and Authentication
Before initiating any Terraform configuration, several foundational requirements must be met to ensure a successful handshake between the Terraform CLI and the Elastic Cloud API.
Terraform Version Requirements
A minimum version of Terraform 1.2.7 is required for full compatibility with current provider features. Using an outdated version of the CLI can lead to syntax errors or failure to recognize newer resource attributes.
Account Requirements
Users must possess an active Elastic Cloud account associated with cloud.elastic.co. This account serves as the administrative root for all deployments managed via the provider.
The API Key Mechanism
Authentication is handled via an API key. A critical distinction must be made here: the key required is an Elastic Cloud Account API key, not an API key generated from within a specific Elasticsearch instance. The account-level key provides the permissions necessary to create and destroy entire deployments, whereas an instance key is limited to data operations within a single cluster.
To generate the required key:
1. Log in to the cloud.elastic.co console.
2. Access the user menu located in the top right of the interface.
3. Select the Organization option.
4. Navigate to the API keys section.
5. Click Create API key.
6. Immediately copy the key, as it is displayed only once for security reasons.
Provider Declaration and Configuration
The initialization of the Terraform environment begins with the declaration of the providers and their respective versions. This ensures that every member of a team is using the same provider version, preventing unexpected behavior during terraform apply.
The following configuration represents a standard setup in a versions.tf file:
hcl
terraform {
required_version = ">= 1.2.7"
required_providers {
ec = {
source = "elastic/ec"
version = "~> 0.13"
}
elasticstack = {
source = "elastic/elasticstack"
version = "~> 0.16"
}
}
}
Once the providers are declared, the ec provider must be configured with the API key. There are two primary methods for achieving this.
Method 1: Variable-based Configuration
In this approach, the API key is passed as a variable, which is marked as sensitive to prevent the key from appearing in plain text within logs.
```hcl
provider "ec" {
apikey = var.ecapikey
}
variable "ecapikey" {
type = string
sensitive = true
description = "Elastic Cloud API key"
}
```
Method 2: Environment Variable Configuration
For those integrating with CI/CD pipelines (like GitHub Actions or GitLab CI), using environment variables is the preferred method for security and flexibility.
bash
export EC_API_KEY="your-elastic-cloud-api-key"
When the environment variable EC_API_KEY is set, the provider is configured to automatically pick up the key, allowing the provider block in the HCL file to remain empty:
hcl
provider "ec" {
}
Deploying an Elastic Cloud Instance
The core resource for provisioning is the ec_deployment. This resource allows for the precise definition of the cluster's geographical location, the version of the software stack, and the hardware template.
A minimal configuration to bring up a functional cluster is as follows:
hcl
resource "ec_deployment" "custom-deployment-id" {
name = "My deployment identifier"
region = "gcp-europe-west3"
version = "8.1.3"
deployment_template_id = "gcp-memory-optimized-v2"
elasticsearch {}
kibana {}
}
Detailed Breakdown of Resource Arguments:
- name: A unique identifier for the deployment within the Elastic Cloud console.
- region: The specific cloud provider and data center (e.g.,
gcp-europe-west3). This impacts latency and data residency compliance. - version: The specific version of the Elastic Stack. Pinning the version is a best practice to avoid unexpected disruptions caused by automatic upgrades.
- deploymenttemplateid: A predefined set of resource allocations (CPU, RAM, Storage) optimized for specific workloads, such as
gcp-memory-optimized-v2. - elasticsearch {}: A block that tells the provider to provision the Elasticsearch search engine.
- kibana {}: A block that provisions the Kibana visualization interface.
Execution Workflow
To deploy the above configuration, the user must follow a three-step terminal sequence:
terraform init: Initializes the working directory and downloads the required providers.terraform validate: Ensures the configuration is syntactically correct.terraform apply -auto-approve: Executes the plan. The actual provisioning process typically takes between 1 and 3 minutes while the cloud provider spins up the Kibana and Elasticsearch nodes.
Managing Outputs and Connectivity
Once the deployment is successful, Terraform can capture the dynamic endpoints generated by Elastic Cloud and output them for use by other applications or for manual verification.
```hcl
output "elasticsearchendpoint" {
value = ecdeployment.custom-deployment-id.elasticsearch[0].https_endpoint
}
output "elasticsearchusername" {
value = ecdeployment.custom-deployment-id.elasticsearch_username
}
output "elasticsearchpassword" {
value = ecdeployment.custom-deployment-id.elasticsearch_password
sensitive = true
}
output "kibanaendpoint" {
value = ecdeployment.custom-deployment-id.kibana[0].https_endpoint
}
```
The use of sensitive = true for the password output is mandatory to ensure that the plain-text password is not printed to the console during the apply process, protecting the cluster from unauthorized access.
Advanced Architectural Strategies
For production-grade deployments, a simple single-node cluster is rarely sufficient. The Elastic Cloud Terraform provider supports complex architectural patterns to optimize for cost and performance.
Hot-Warm-Cold Architecture
For logging and time-series data, implementing a tiered storage strategy is essential. This is achieved by defining different node roles:
- Hot Nodes: High-performance SSDs for indexing and searching recent data.
- Warm Nodes: Cheaper storage for data that is accessed less frequently.
- Cold Nodes: The most cost-effective storage for long-term retention and compliance data.
Network Security and Traffic Filtering
Deployments should never be left open to the public internet unless absolutely necessary. The provider allows the application of traffic filters, which act as a firewall to restrict access to specific IP ranges or VPCs.
Autoscaling and Resource Management
One of the most powerful features of the Elastic Cloud provider is the ability to allow Elastic Cloud to adjust resources based on actual usage. This ensures that the cluster can handle traffic spikes without manual intervention and scale down during idle periods to save costs.
Version Management
To avoid the "dependency hell" of unexpected upgrades, it is recommended to use the data.ec_stack data source. This allows Terraform to dynamically discover available versions of the Elastic Stack and apply them in a controlled manner.
AWS Integration and Data Migration
In specific AWS-centric environments, Terraform modules can be used to automate the entire ecosystem surrounding an Elastic Cloud deployment. This extends beyond the Elastic provider to include AWS provider resources.
The following components are typically provisioned in a comprehensive AWS-Elastic integration:
- Elastic Cloud Cluster: The core search and analytics engine.
- Amazon EC2 Instances: Necessary for running Elastic Agents that collect data from the network.
- Amazon S3 Buckets: Used as the destination for Elasticsearch snapshots to ensure disaster recovery capabilities.
- Elastic Serverless Forwarder: A specialized tool used for efficient data ingestion into the Elastic Cloud.
- IAM Instance Roles: Fine-grained permissions allowing EC2 instances to interact with S3 and other AWS services securely without storing hardcoded credentials.
For organizations migrating from on-premises, the Terraform workflow can be extended to include a data migration process, moving data from a self-managed Elasticsearch cluster directly into the newly provisioned AWS Elastic Cloud instance.
Comparative Provider Specifications
| Feature | Elastic Cloud (ec) Provider | Elastic Stack (elasticstack) Provider |
|---|---|---|
| Primary Purpose | Infrastructure Provisioning | Resource Configuration |
| Management Target | Clusters, Projects, Deployments | Indices, Users, Product Settings |
| API Target | Elastic Cloud Management API | Elasticsearch/Kibana APIs |
| Typical Use Case | Creating a 3-node cluster in AWS | Creating an index template |
| Versioning Focus | Deployment Template & Stack Version | Internal Product Versioning |
| Scope | Infrastructure-as-Code (IaC) | Configuration-as-Code (CaC) |
Operational Best Practices
To maintain a stable and secure Elastic Cloud environment, the following operational guidelines should be strictly followed:
- Isolation of Environments: Use entirely separate deployments for development, staging, and production. Sharing a single cluster across environments increases the blast radius of a configuration error.
- Secret Management: Sensitive settings should be stored in the Elastic keystore rather than being passed as plain-text strings within the Terraform configuration files.
- Dependency Tracking: Always consistently utilize the latest versions of both the Elastic Cloud Terraform provider and the Terraform CLI to ensure access to the latest bug fixes and feature sets.
- Lifecycle Monitoring: While Terraform manages the deployment, ongoing health monitoring should be handled by observability tools. Integration with platforms like OneUptime can provide a unified view of the Elastic deployment alongside the rest of the corporate infrastructure.
Detailed Analysis of Provider Evolution
The Elastic Cloud Terraform provider is currently in a pre-1.0.0 state. This is a significant detail for DevOps engineers because it implies that model changes may be introduced between minor versions. Until version 1.0.0 is officially released, users must rely heavily on the change log and individual release notes to understand how a provider update might impact their existing infrastructure.
The trajectory of the provider is moving toward a unified orchestration lifecycle. By providing a common set of APIs across Elastic Serverless (ESS), Elastic Serverless Projects (ESSP), and Elastic Cloud Enterprise (ECE), Elastic is attempting to create a seamless experience where the underlying hosting model is abstracted away from the Terraform configuration. This means that a user could potentially move a workload from a hosted environment to a self-managed ECE environment with minimal changes to their HCL code, provided they update the provider endpoint.
This evolution reflects a broader trend in the industry toward "platform engineering," where the goal is to provide a curated internal developer platform. By exposing the Elastic Cloud capabilities through Terraform, Elastic is enabling platform teams to create "golden paths" for their developers—pre-approved, secure, and right-sized Elastic clusters that can be deployed in seconds via a simple terraform apply command.