Running Terraform against AWS without internet access requires private connectivity to AWS services. VPC Endpoints are the backbone for that connectivity. They use AWS PrivateLink and Elastic Network Interfaces to route traffic inside the AWS network. In an air-gapped VPC Terraform can still run terraform init and apply state when S3, DynamoDB, and interface endpoints for control plane services are correctly configured.

Introduction

Terraform in an air-gapped AWS environment depends on VPC Endpoints. The previous guide covered pre-downloading providers and modules. This article focuses on running Terraform in an air-gapped AWS environment and how to leverage VPC Endpoints to enable secure communication with AWS services without relying on internet connectivity.

When Terraform operates in an AWS environment isolated from the public internet, VPC Endpoints act as the backbone, allowing Terraform to interact privately with AWS services. This guide explores the essential VPC endpoints required for Terraform and provides a step-by-step process to configure them.

The guide is at a beginner level difficulty.

What Are VPC Endpoints

VPC Endpoints allow AWS resources in your VPC to privately connect to AWS services without going over the public internet. They use the AWS PrivateLink service and Elastic Network Interfaces to route traffic securely through the AWS network.

There are two types of VPC Endpoints:

Gateway Endpoints: For S3 and DynamoDB, free and highly efficient
Interface Endpoints: For all other services like EC2, IAM, STS, RDS, etc., incur costs

Step-by-Step Guide to Configuring VPC Endpoints

S3 Gateway Endpoint

The S3 Gateway Endpoint is required for storing Terraform state files in S3 and downloading modules or remote files from an S3 bucket.

Create the S3 Gateway Endpoint:

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.s3 \ --route-table-ids <route-table-id>

Route Table: Add the endpoint to the route table associated with your private subnets.

Permissions: Restrict access to specific buckets using an endpoint policy:

json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws:s3:::my-terraform-state-bucket", "arn:aws:s3:::my-terraform-state-bucket/*" ] } ] }

DynamoDB Gateway Endpoint

The DynamoDB Gateway Endpoint is required if you are using DynamoDB for state locking.

Create the DynamoDB Gateway Endpoint:

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.dynamodb \ --route-table-ids <route-table-id>

No additional endpoint policies are needed for DynamoDB.

Interface Endpoints for Terraform-Managed AWS Services

Terraform needs interface VPC Endpoints for most AWS services. Below are the key services and how to configure them.

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.ec2 \ --subnet-ids <subnet-id> \ --security-group-ids <security-group-id>

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.iam \ --subnet-ids <subnet-id> \ --security-group-ids <security-group-id>

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.sts \ --subnet-ids <subnet-id> \ --security-group-ids <security-group-id>

bash aws ec2 create-vpc-endpoint \ --vpc-id <vpc-id> \ --service-name com.amazonaws.<region>.secretsmanager \ --subnet-ids <subnet-id> \ --security-group-ids <security-group-id>

Configure Security Groups for VPC Endpoints

For interface VPC Endpoints, ensure the associated security groups allow incoming traffic from the private subnet CIDRs.

Example Security Group Rules:
- Allow HTTPS port 443 traffic from private subnet CIDRs

The Terraform AWS VPC Endpoint Module is designed to create VPC endpoints on an existing VPC in your AWS infrastructure. This module offers the ability to automatically generate a dedicated security group for all Interface endpoints when the createendpointsg variable is set to true, we recommend this setting to be set.

A skipped Checkov alert CKV2AWS5 in the complete example of the Terraform module for VPC endpoints is intentionally skipped as VPC endpoints are configured to utilize security groups only for specific Interface VPC endpoints. The security group attached is specifically designed to allow SSL/TLS inbound traffic on port 443.

Enable Private DNS for Interface Endpoints

Enable private DNS names so that Terraform can use the standard AWS service URLs e.g., ec2.amazonaws.com without modification.

bash aws ec2 modify-vpc-endpoint \ --vpc-endpoint-id <endpoint-id> \ --private-dns-enabled

Validating the Setup

After creating the necessary VPC Endpoints:

Run terraform init and ensure it can access the S3 bucket and DynamoDB table
Deploy a sample resource e.g., EC2 instance or IAM role to verify connectivity to the respective AWS services

Running Terraform in an air-gapped AWS environment requires thoughtful planning and VPC Endpoint configurations. By setting up the necessary endpoints S3, DynamoDB, EC2, IAM, etc., you enable Terraform to interact with AWS services securely without internet access.

Terraform Modules for VPC Endpoints

Boldlink VPC Endpoints Module

This module is easy to use with simplified examples, removes the complexity of managing multiple resources manually, and deploys the needed resources faster.

Example minimum VPC endpoints:

hcl module "minimum_vpc_endpoints" { source = "boldlink/vpc-endpoints/aws/" version = "<latest_version_nr>" vpc_id = local.vpc_id tags = var.tags vpc_endpoints = [ { service_name = "com.amazonaws.${local.region}.dynamodb" vpc_endpoint_type = "Gateway" name = "DynamoDB" route_table_ids = flatten(local.route_table_ids) policy = data.aws_iam_policy_document.ddb_endpoint_policy.json } ] }

Charges apply. See here for more details.

Infrablocks VPC Endpoint Module

A Terraform module for managing a VPC endpoint in AWS.

To use the module, include something like the following in your Terraform configuration:

hcl module "vpc_endpoint" { source = "infrablocks/vpc-endpoint/aws" version = "0.0.0" }

See the Terraform registry entry for more details.

This module is compatible with Terraform versions greater than or equal to Terraform 1.3.

In order for the build to run correctly, a few tools will need to be installed on your development machine:

Ruby 3.1.1
Bundler
git
git-crypt
gnupg
direnv
aws-vault

Installing the required tools is best managed by homebrew.

To install homebrew:

bash ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then, to install the required tools:

```bash

ruby

brew install rbenv
brew install ruby-build
echo 'eval "$(rbenv init - bash)"' >> ~/.bash_profile
echo 'eval "$(rbenv init - zsh)"' >> ~/.zshrc
eval "$(rbenv init -)"
rbenv install 3.1.1
rbenv rehash
rbenv local 3.1.1
gem install bundler

git, git-crypt, gnupg

brew install git
brew install git-crypt
brew install gnupg

aws-vault

brew cask install

direnv

brew install direnv
echo "$(direnv hook bash)" >> ~/.bash_profile
echo "$(direnv hook zsh)" >> ~/.zshrc
eval "$(direnv hook $SHELL)"
direnv allow
```

Running the build requires an...

The API gateway requires:
- TODO

The API gateway consists of:
- TODO

Module parameters are documented in the source with a table of Name, Description, Default, Required.

Name	Description	Default	Required
Name	Description
---	---

Private Hosted Zone and Shared Endpoints

Sometimes instead of a simple AWS VPC Endpoint, you need to control your own DNS through a Private Hosted Zone. This is useful when you have multiple VPCs that need to use the same VPC Endpoints as you cannot share the AWS created DNS across VPCs/Accounts. In this guide, we walk through how to set up a VPC Endpoint with a Private Hosted Zone using Terraform.

Challenge

AWS VPC Endpoint DNS is typically limited to the originating VPC. This means that if you have multiple VPCs that need to access the same VPC Endpoint, you cannot share the DNS resolution across VPCs. This is a problem if you have a multi-account or multi-VPC setup where you want to share a VPC Endpoint across VPCs.

When you create a VPC Endpoint, and turn off the privatednsenabled flag, you lose the ability to use the AWS provided DNS. You also are not given information about what DNS records you should have created in your Private Hosted Zone. Making assumptions here leads to a lot of misses and misconfigurations. Fortunately, the data is available in the AWS API, and we can use Terraform to extract it.

Solution

Using awsvpcendpointservice to create a dynamic configuration that handles all the multitude of DNS entries is a solution where we remove the guess work from this process. This is a relatively new ability, as the data object for this resource was missing the required privatedns_names until recently. Let’s break it down step by step.

VPC Endpoint Types Comparison

Endpoint Type	Services	Cost	Connection
Gateway	S3, DynamoDB	Free	Route table
Interface	EC2, IAM, STS, RDS, SecretsManager, etc.	Incur costs	ENI, PrivateLink

Conclusion

This setup ensures a secure, compliant, and efficient workflow, even in the most restrictive environments.

All code in this post can be found on my GitHub. I’ll be posting more of these guides, going over a wide range of difficulty, so subscribe below. My newsletter sends out friendly emails when I make new posts.

Want to learn more about how I can assist you with your cloud and DevOps needs? Visit my homepage to get in touch and let’s find out how I can support your next project.

AWS VPC Endpoint Terraform for Air-Gapped and Private Connectivity