Orchestrating Secure Connectivity: Configuring aws_vpc_endpoint in Terraform for Air-Gapped and Multi-Account Architectures

In modern cloud infrastructure, the aws_vpc_endpoint resource in Terraform serves as a critical mechanism for decoupling internal workloads from the public internet while maintaining seamless communication with AWS services. For organizations operating in highly regulated environments, such as financial institutions, healthcare providers, or government agencies, the ability to run infrastructure-as-code tools like Terraform within an air-gapped or isolated Virtual Private Cloud (VPC) is not merely a convenience but a compliance requirement. The core challenge in these environments is that Terraform requires access to various AWS APIs—such as EC2, IAM, STS, S3, and DynamoDB—to function correctly. Without direct internet egress, these calls fail. VPC Endpoints resolve this by providing private, secure pathways through the AWS network, utilizing AWS PrivateLink and Elastic Network Interfaces (ENIs) to route traffic. This article provides a comprehensive technical deep dive into configuring aws_vpc_endpoint resources, detailing the differences between Gateway and Interface endpoints, the implementation of security policies, the management of private DNS records, and advanced techniques for multi-account architectures using Private Hosted Zones.

Fundamental Architecture of VPC Endpoints in Terraform

To understand the implementation of aws_vpc_endpoint, one must first distinguish between the two primary types of endpoints: Gateway Endpoints and Interface Endpoints. This distinction dictates the underlying network mechanics, cost implications, and the specific Terraform attributes required for configuration.

Gateway Endpoints are designed for specific services, primarily Amazon S3 and Amazon DynamoDB. They are distinct from Interface Endpoints in that they do not create Elastic Network Interfaces (ENIs) within the VPC. Instead, they rely on route table entries to direct traffic to the AWS service. This architectural difference makes Gateway Endpoints highly efficient and, crucially, free of charge. In a Terraform context, configuring a Gateway Endpoint involves specifying the VPC ID, the service name, and the associated route table IDs. Because these endpoints do not incur hourly costs or data processing charges, they are the preferred method for accessing S3 and DynamoDB, especially when the volume of state file operations or state locking is high.

Interface Endpoints, conversely, are available for nearly all other AWS services, including EC2, IAM, STS, Secrets Manager, RDS, and Lambda. These endpoints create an ENI in the subnets specified within the VPC. The ENI holds a private IP address, and traffic destined for the AWS service is routed to this ENI, which then forwards the traffic over the AWS backbone to the service. Interface Endpoints incur costs, typically consisting of an hourly fee for the endpoint itself and a data processing charge. Therefore, when configuring Terraform infrastructure, it is essential to map the required services against their endpoint types to optimize costs.

The following table summarizes the key differences between Gateway and Interface endpoints, which dictates the Terraform configuration strategy:

Configuring Gateway Endpoints for State Management

The foundational requirement for running Terraform in an isolated environment is the ability to store and lock the Terraform state. By default, Terraform stores state locally, which is unsuitable for team collaboration or remote execution. The standard practice is to use an S3 bucket for state storage and DynamoDB for state locking to prevent concurrent write conflicts. Both of these services are best accessed via Gateway Endpoints due to their zero-cost nature.

S3 Gateway Endpoint Configuration

The S3 Gateway Endpoint is mandatory for storing Terraform state files and downloading modules or remote files from S3 buckets. In Terraform, this is implemented using the aws_vpc_endpoint resource with the vpc_endpoint_type set to "Gateway". The configuration requires the vpc_id and the route_table_ids associated with the private subnets where the Terraform execution environment resides.

It is critical to restrict access to specific buckets using an endpoint policy. Allowing unrestricted access to S3 through the endpoint undermines the security posture of the air-gapped environment. The following code block demonstrates the Terraform configuration for an S3 Gateway Endpoint with a restrictive endpoint policy:

```hcl
resource "awsvpcendpoint" "s3" {
vpcid = var.vpcid
servicename = "com.amazonaws.${var.region}.s3"
vpcendpoint_type = "Gateway"

routetableids = var.privateroutetable_ids

policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = ""
Action = "s3:"
Resource = [
"arn:aws:s3:::my-terraform-state-bucket",
"arn:aws:s3:::my-terraform-state-bucket/*"
]
}
]
})
}
```

The endpoint policy above ensures that only the specific S3 bucket designated for Terraform state can be accessed. Any attempt to access other S3 resources through this endpoint will be denied. This least-privilege approach is vital in air-gapped environments where the perimeter is strictly controlled.

DynamoDB Gateway Endpoint Configuration

If the Terraform state backend utilizes DynamoDB for locking (a common practice to prevent race conditions during terraform apply or terraform plan), a DynamoDB Gateway Endpoint must be configured. Unlike S3, DynamoDB does not require a complex endpoint policy for basic functionality, and no additional endpoint policies are strictly needed for standard state locking operations.

The Terraform configuration for the DynamoDB endpoint mirrors the S3 setup but targets the DynamoDB service. It is essential to ensure that the route table IDs match those of the subnets where the Terraform runner is located.

```hcl
resource "awsvpcendpoint" "dynamodb" {
vpcid = var.vpcid
servicename = "com.amazonaws.${var.region}.dynamodb"
vpcendpoint_type = "Gateway"

routetableids = var.privateroutetable_ids
}
```

By configuring these Gateway Endpoints, the Terraform CLI can interact with S3 and DynamoDB without any traffic traversing the public internet. The route table entries ensure that any traffic destined for the S3 or DynamoDB service IPs is intercepted and routed directly to the AWS service infrastructure via the endpoint.

Implementing Interface Endpoints for API Connectivity

While Gateway Endpoints handle state management, Terraform requires Interface Endpoints to interact with the majority of AWS services. When Terraform provisions resources, it must call APIs for services such as EC2 to create instances, IAM to manage users and roles, and STS to retrieve temporary credentials. In an air-gapped VPC, these API calls must be routed through Interface Endpoints.

Interface Endpoints are more complex to configure than Gateway Endpoints because they require the specification of subnets and security groups. The ENIs created by Interface Endpoints reside in the specified subnets and must have security groups attached that permit the necessary traffic. Typically, the security group must allow inbound HTTPS (port 443) traffic from the CIDR blocks of the private subnets where the Terraform execution environment is located.

Essential Interface Endpoints for Terraform

The following table lists the critical AWS services that require Interface Endpoints for Terraform operation in an air-gapped environment, along with their corresponding service name patterns.

The following code block illustrates the Terraform configuration for the essential Interface Endpoints (EC2, IAM, and STS). Note the use of subnet_ids and security_group_ids.

```hcl
resource "awsvpcendpoint" "ec2" {
vpcid = var.vpcid
servicename = "com.amazonaws.${var.region}.ec2"
vpcendpoint_type = "Interface"

subnetids = var.privatesubnetids
securitygroupids = [var.vpcesecuritygroupid]
privatednsenabled = true
}

resource "awsvpcendpoint" "iam" {
vpcid = var.vpcid
servicename = "com.amazonaws.${var.region}.iam"
vpcendpoint_type = "Interface"

subnetids = var.privatesubnetids
securitygroupids = [var.vpcesecuritygroupid]
privatednsenabled = true
}

resource "awsvpcendpoint" "sts" {
vpcid = var.vpcid
servicename = "com.amazonaws.${var.region}.sts"
vpcendpoint_type = "Interface"

subnetids = var.privatesubnetids
securitygroupids = [var.vpcesecuritygroupid]
privatednsenabled = true
}
```

Security Group Configuration

The security group associated with Interface Endpoints plays a pivotal role in the security posture of the VPC. The security group must be configured to allow incoming traffic from the private subnet CIDRs on port 443 (HTTPS). It is a best practice to create a dedicated security group for all Interface Endpoints. When using the Terraform AWS VPC Endpoint module, there is often a variable such as create_endpoint_sg that, when set to true, automatically generates a dedicated security group. This module is recommended for simplifying the management of multiple resources and ensuring that the security group is correctly configured to allow SSL/TLS inbound traffic on port 443.

A common compliance check, such as Checkov alert CKV2_AWS_5, may flag configurations if security groups are not strictly applied to Interface endpoints. However, if the security group is specifically designed to allow SSL/TLS inbound traffic only from trusted internal CIDRs, this alert can be intentionally skipped or handled through policy exceptions. The key is to ensure that the ENIs created by the endpoints are not exposed to the public internet or untrusted internal networks.

Private DNS Resolution and Private Hosted Zones

A significant aspect of VPC Endpoint configuration is the handling of DNS resolution. By default, AWS provides private DNS names for VPC Endpoints, allowing resources within the VPC to access services using standard public endpoints (e.g., ec2.amazonaws.com). This behavior is controlled by the private_dns_enabled attribute in Terraform.

When private_dns_enabled is set to true, the VPC Endpoint creates a private DNS record that resolves the standard AWS service hostname to the private IP address of the ENI. This allows Terraform to use its standard configuration without modification. For example, if private_dns_enabled is true for the EC2 endpoint, the hostname ec2.amazonaws.com will resolve to the private IP of the EC2 VPC Endpoint ENI within the VPC.

However, this default behavior is limited to the originating VPC. In multi-VPC or multi-account architectures, AWS-created private DNS records are not shared across VPCs or accounts. This limitation poses a challenge when multiple VPCs need to access the same VPC Endpoint. In such scenarios, the AWS-provided DNS resolution cannot be shared, leading to a situation where other VPCs cannot resolve the endpoint's private IP address using the standard service hostname.

The Private Hosted Zone (PHZ) Solution

To address the multi-VPC DNS challenge, administrators can disable the AWS-provided private DNS (private_dns_enabled = false) and instead manage DNS resolution through a Private Hosted Zone (PHZ). This approach requires extracting the DNS record information from the AWS API and creating corresponding records in the PHZ.

The aws_vpc_endpoint_service data source in Terraform provides access to the private_dns_names of the endpoint. By utilizing this data source, Terraform can dynamically construct the necessary DNS entries in the Private Hosted Zone. This removes the guesswork involved in manually configuring DNS records, which is a common source of misconfiguration when making assumptions about the DNS structure.

The following code block demonstrates how to use the aws_vpc_endpoint_service data source to retrieve DNS names and create records in a Private Hosted Zone:

```hcl
data "awsvpcendpointservice" "ec2" {
servicename = "com.amazonaws.${var.region}.ec2"
}

resource "awsroute53zone" "privatezone" {
name = "example.com"
vpc {
vpcid = var.vpcid
zoneid = var.privatezoneid
}
}

resource "awsroute53record" "ec2endpoint" {
zoneid = awsroute53zone.private_zone.id
name = "ec2.amazonaws.com"
type = "A"
ttl = 300

# This list will contain the private IP addresses of the ENIs created by the VPC Endpoint
records = data.awsvpcendpointservice.ec2.privatedns_names
}
```

This configuration ensures that any VPC associated with the Private Hosted Zone can resolve ec2.amazonaws.com to the private IP addresses of the VPC Endpoint ENIs, effectively sharing the endpoint's connectivity across multiple VPCs. This technique is particularly useful in large-scale enterprise environments where centralizing VPC Endpoints in a dedicated network VPC and sharing them across application VPCs is a common architectural pattern.

Modularization and Best Practices

Managing VPC Endpoints individually can become cumbersome in large-scale infrastructure. The Terraform AWS VPC Endpoint module provides a standardized approach to creating and managing VPC Endpoints. This module simplifies the configuration process by abstracting the underlying resources and providing a consistent interface for defining endpoints.

The module offers the ability to automatically generate a dedicated security group for all Interface endpoints when the create_endpoint_sg variable is set to true. This is a recommended setting to ensure that security best practices are followed without manual intervention. The module also handles the complexity of managing multiple resources, such as the endpoints themselves, security groups, and potentially DNS records.

A simplified example of using the module is shown below:

```hcl
module "minimumvpcendpoints" {
source = "boldlink/vpc-endpoints/aws/"
version = ""

vpcid = local.vpcid
tags = var.tags

vpcendpoints = [
{
servicename = "com.amazonaws.${local.region}.dynamodb"
vpcendpointtype = "Gateway"
name = "DynamoDB"
routetableids = flatten(local.routetableids)
policy = data.awsiampolicydocument.ddbendpoint_policy.json
}
]
}
```

Using modules not only reduces code duplication but also ensures that all VPC Endpoints are configured consistently across the environment. It also facilitates easier maintenance and updates, as changes to the module are reflected in all environments using it.

Validating the Setup

After deploying the VPC Endpoints, it is crucial to validate the configuration to ensure that Terraform can successfully communicate with the AWS services. The validation process typically involves the following steps:

1. Run terraform init in the directory containing the Terraform configuration. This step will attempt to connect to the S3 backend. If the S3 Gateway Endpoint is correctly configured, this command should succeed without internet access.
2. Run terraform plan to generate an execution plan. This step will exercise the Interface Endpoints for EC2, IAM, and STS. If the plan completes successfully, it indicates that the necessary API calls are being routed through the VPC Endpoints.
3. Deploy a sample resource, such as an EC2 instance or an IAM role, to verify connectivity. This step ensures that the full lifecycle of resource creation works through the private endpoints.

If terraform init fails, it is likely due to a misconfigured S3 Gateway Endpoint or an incorrect route table association. If terraform plan fails, it may indicate issues with the Interface Endpoints for EC2, IAM, or STS, such as incorrect security group rules or missing private DNS records.

Conclusion

Configuring aws_vpc_endpoint in Terraform is a foundational skill for building secure, compliant, and efficient cloud infrastructure. By leveraging Gateway Endpoints for S3 and DynamoDB, organizations can manage Terraform state without incurring costs or relying on the public internet. Interface Endpoints provide the necessary private connectivity for interacting with the vast majority of AWS services, ensuring that API calls are routed securely through the AWS network. The ability to manage private DNS records and utilize Private Hosted Zones further enhances the flexibility of VPC Endpoints, enabling shared access across multiple VPCs and accounts. By adopting best practices such as using dedicated security groups, implementing restrictive endpoint policies, and utilizing Terraform modules, organizations can create a robust and secure environment for running Terraform in air-gapped or isolated AWS setups. This setup not only meets compliance requirements but also improves the performance and reliability of infrastructure-as-code workflows by reducing latency and eliminating the risks associated with public internet exposure.

Sources

1. Running Terraform in an Air-Gapped Environment — Part 2
2. Terraform Foundation terraform-aws-vpc-endpoints
3. Terraform VPC Endpoints with Private Hosted Zone
4. Creating AWS VPC Endpoints with Terraform: A Step-by-Step Guide