Architecting Private AWS Connectivity with terraform aws_vpc_endpoint

In modern cloud architecture, security is often synonymous with isolation. For organizations operating in highly regulated industries or those employing a strict "air-gapped" philosophy, allowing traffic to traverse the public internet—even when encrypted—is an unacceptable risk. This is where the AWS VPC Endpoint becomes a critical architectural component. When combined with Terraform, the industry-standard Infrastructure as Code (IaC) tool, engineers can programmatically define a perimeter that allows private communication between a Virtual Private Cloud (VPC) and AWS services.

A VPC Endpoint enables private connectivity between your VPC and supported AWS services without requiring an internet gateway, NAT device, VPN connection, or Direct Connect. This is achieved through the AWS PrivateLink service, which creates an Elastic Network Interface (ENI) with a private IP address from your subnet. For the Terraform operator, this means that terraform init, terraform apply, and terraform destroy can be executed from a build server locked inside a private subnet, interacting with S3 for state storage and DynamoDB for state locking without ever exposing a single packet to the public web.

Taxonomy of VPC Endpoints

To effectively implement aws_vpc_endpoint in Terraform, one must first understand the two distinct categories of endpoints available. These differ significantly in terms of cost, routing mechanisms, and the services they support.

Gateway Endpoints

Gateway Endpoints are specialized targets used specifically for Amazon S3 and Amazon DynamoDB. Unlike other endpoints, they do not use ENIs; instead, they are targets in your VPC route table.

• Cost Efficiency: Gateway Endpoints are free of charge.
• Routing: They function as a routing target. When you create a Gateway Endpoint, you associate it with one or more route tables. AWS then adds a route to that table directing traffic destined for the S3 or DynamoDB prefix list to the endpoint.
• Primary Use Case: Essential for Terraform backends utilizing S3 for state files and DynamoDB for state locking in isolated environments.

Interface Endpoints

Interface Endpoints are the general-purpose solution for the vast majority of AWS services (e.g., EC2, IAM, STS, RDS, Secrets Manager). They are powered by AWS PrivateLink.

• Implementation: They place an ENI (Elastic Network Interface) directly into your subnet.
• Cost: Interface endpoints incur hourly charges and data processing fees.
• DNS Integration: They can be configured with "Private DNS," allowing the resource to use standard AWS service URLs (such as ec2.amazonaws.com) while resolving them to the private IP of the endpoint.

Summary Comparison Table

Implementing Terraform in Air-Gapped AWS Environments

Running Terraform in an air-gapped environment requires a meticulous approach to connectivity. Since the terraform binary cannot reach the public internet to download providers or communicate with the AWS API, the VPC must be equipped with specific endpoints to facilitate these operations.

S3 Gateway Endpoint Configuration

The S3 Gateway Endpoint is the most critical piece of infrastructure for a Terraform-managed environment, as it is required for storing state files and downloading remote modules.

To create an S3 Gateway Endpoint using the AWS CLI, the following command is used:

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.s3 \ --route-table-ids <route-table-id>

From a security perspective, it is recommended to restrict access using an endpoint policy. This ensures that the endpoint cannot be used to exfiltrate data to unauthorized S3 buckets. An example policy for a Terraform state bucket is provided below:

json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws:s3:::my-terraform-state-bucket", "arn:aws:s3:::my-terraform-state-bucket/*" ] } ] }

DynamoDB Gateway Endpoint Configuration

If your Terraform backend configuration utilizes a DynamoDB table for state locking (preventing concurrent executions from corrupting the state file), a DynamoDB Gateway Endpoint is required.

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.dynamodb \ --route-table-ids <route-table-id>

Unlike S3, DynamoDB Gateway Endpoints typically do not require additional endpoint policies for standard Terraform operations.

Essential Interface Endpoints for Terraform Management

While Gateway endpoints handle state and locking, Terraform needs Interface endpoints to actually manage AWS resources (the "Plan" and "Apply" phases). The following services are typically mandatory:

• EC2: For managing instances, security groups, and networking.
• IAM: For managing roles and policies.
• STS: For assuming roles across accounts or regions.
• Secrets Manager: For retrieving sensitive variables securely.

Example CLI implementation for an Interface Endpoint:

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.ec2 \ --subnet-ids <subnet-id> \ --security-group-ids <security-group-id>

Advanced Configuration and Security

Properly configuring an aws_vpc_endpoint requires more than just the creation of the resource; it requires the orchestration of security groups and DNS settings.

Security Group Orchestration

Interface Endpoints are governed by security groups. To ensure Terraform can communicate with the endpoint, the associated security group must allow inbound traffic from the private subnet CIDRs where the Terraform runner is located. Specifically, SSL/TLS traffic on port 443 must be permitted.

In a professional Terraform module implementation, it is highly recommended to set create_endpoint_sg = true. This allows the module to automatically generate a dedicated security group tailored for the Interface endpoints, reducing the risk of manual configuration errors and ensuring that only port 443 is open for inbound traffic.

Private DNS Resolution

By default, an interface endpoint provides a DNS name that is unique to that endpoint. However, most software (including the AWS SDKs used by Terraform) is configured to call the standard service URL (e.g., ec2.amazonaws.com).

To resolve this, "Private DNS" must be enabled. This allows the VPC's internal DNS to intercept requests for the public service URL and route them to the private IP of the VPC endpoint.

bash aws ec2 modify-vpc-endpoint \ --vpc-endpoint-id <endpoint-id> \ --private-dns-enabled

Solving Multi-VPC and Multi-Account DNS Challenges

A significant challenge arises when multiple VPCs need to share a single set of VPC Endpoints to save on costs. Because AWS-created Private DNS is limited to the VPC where the endpoint is located, other VPCs cannot resolve those names.

The Private Hosted Zone (PHZ) Solution

When private_dns_enabled is set to false, the developer loses the ability to use AWS-provided DNS. To solve this across multiple VPCs, you must implement a Private Hosted Zone (PHZ) in Route 53.

The difficulty lies in knowing exactly which DNS records to create in the PHZ. Rather than guessing or manually searching for records, an expert approach uses Terraform to extract the required private_dns_names from the aws_vpc_endpoint_service data object. This dynamic configuration ensures that as AWS updates the endpoint's underlying DNS, the PHZ remains synchronized, allowing a centralized "Shared Services" VPC to provide connectivity for the entire organization.

Cross-Region VPC Endpoint Connectivity

Connecting a VPC Endpoint Service in one region to a VPC Endpoint in another region introduces a specific Terraform configuration pitfall.

The "InvalidServiceName" Error

When attempting to create an aws_vpc_endpoint in a different region than the service provider, engineers often encounter the following error:
Error: creating EC2 VPC Endpoint... api error InvalidServiceName: The Vpc Endpoint Service... does not exist.

This occurs because Terraform, by default, looks for the service in the region defined by the provider. For example, if you are creating a VPC endpoint in Oregon (us-west-2) that connects to a service in Ohio (us-east-2), Terraform will search for the service in Oregon and fail.

Resolution through service_region

To resolve this, you must explicitly define the service_region within the aws_vpc_endpoint resource. This tells Terraform exactly where the service exists, regardless of where the endpoint is being deployed.

Example Configuration:
hcl resource "aws_vpc_endpoint" "test_vpc_endpoint" { provider = aws.oregon vpc_id = var.vpc_id_oregon vpc_endpoint_type = "Interface" service_name = data.aws_vpc_endpoint_service.rds_endpoint_service.service_name # The critical addition for cross-region connectivity: service_region = "us-east-2" security_group_ids = var.glue_security_groups subnet_ids = var.glue_subnets private_dns_enabled = var.glue_private_dns_enabled tags = local.tags depends_on = [aws_vpc_endpoint_service.rds_endpoint_service] }

Tooling and Modules for Implementation

While the aws_vpc_endpoint resource is powerful, managing dozens of endpoints manually is tedious. The community provides modules to streamline this.

Utilizing the boldlink/vpc-endpoints/aws Module

This module simplifies the deployment of multiple endpoints. By defining a list of endpoints in a variable, the module handles the creation of the endpoints and their corresponding security groups.

Example implementation:
hcl module "minimum_vpc_endpoints" { source = "boldlink/vpc-endpoints/aws/" version = "<latest_version_nr>" vpc_id = local.vpc_id tags = var.tags vpc_endpoints = [ { service_name = "com.amazonaws.${local.region}.dynamodb" vpc_endpoint_type = "Gateway" name = "DynamoDB" route_table_ids = flatten(local.route_table_ids) policy = data.aws_iam_policy_document.ddb_endpoint_policy.json } ] }

Local Development Environment Requirements

For developers building or contributing to VPC endpoint modules (such as those found in the infrablocks repository), a specific local toolset is required to ensure build consistency. This environment typically includes:

• Runtime: Ruby (3.1.1) managed via rbenv.
• Package Management: Bundler.
• Version Control: Git.
• Security & Secrets: git-crypt, gnupg, and aws-vault for secure credential management.
• Environment Management: direnv to load environment variables dynamically.

Conclusion

The aws_vpc_endpoint is an indispensable resource for any organization prioritizing security through isolation. Whether you are configuring a simple air-gapped environment for Terraform runners or architecting a complex, multi-region, multi-VPC shared services hub, the nuances of endpoint selection—Gateway vs. Interface—are paramount.

For those operating in air-gapped environments, the synergy between S3 and DynamoDB Gateway endpoints and the broader suite of Interface endpoints (EC2, IAM, STS) creates a secure loop that allows infrastructure to be managed entirely offline from the public internet. The primary technical hurdles—namely DNS resolution in multi-VPC setups and the service_region requirement for cross-region connectivity—can be overcome with a combination of Route 53 Private Hosted Zones and explicit Terraform resource attributes.

By shifting from manual CLI creation to modular Terraform configurations, teams can ensure that security group rules (specifically port 443) are applied consistently, and that the infrastructure remains scalable and reproducible. The ultimate goal of implementing VPC endpoints is to remove the "internet" as a variable in the security equation, turning the AWS cloud into a truly private extension of the corporate data center.

Sources

1. Running Terraform in an Air-Gapped Environment — Part 2
2. The Terraform AWS VPC Endpoint Module
3. A Terraform module for managing a VPC endpoint in AWS
4. Terraform VPC Endpoints with PHZ
5. Set-up a cross-region connection between VPC Endpoint Service in two different regions