AWS Batch Terraform Deployment Patterns and Provider Options

Introduction

AWS Batch removes the operational burden of provisioning and scaling compute for batch workloads by accepting a declarative description of the required resources and handling placement, scaling, and scheduling automatically. When those descriptions are expressed as Terraform code, the entire pipeline of compute environments, job queues, job definitions, and supporting IAM, networking, and scheduling resources becomes repeatable and auditable. The reference material covers two Terraform provider paths for AWS Batch, module design for compute environments, scheduled job workflows, EKS integration, and community module examples for EC2 and Spot compute resources. This article synthesizes those patterns with depth and technical specificity.

AWS Provider Landscape for Batch

Terraform interacts with AWS through two distinct providers.

The original Terraform AWS provider is an open-source project with community-driven pull requests. The provider is hand-coded infrastructure-as-code library that makes calls directly using the AWS SDK, which in turn call AWS APIs. While this approach provides a great developer experience, it can take some time to review and incorporate pull requests that support new AWS service features and new AWS services.

In the middle of 2024, the Terraform AWS Cloud Control provider was made generally available by HashiCorp. This provider works with the AWS Cloud Control API, which is a set of common APIs that make it easy for developers and partners to manage the lifecycle of AWS and third-party services. In contrast to the original AWS Provider, the AWSCC provider is automatically generated based on the Cloud Control API published by AWS. That means the latest features and services from AWS can be supported right away.

Until recently, AWS Batch job definitions weren’t supported by the AWS Cloud Control API as a managed resource. For this reason the AWS Batch team has been focusing on the original AWS provider for providing a blueprint for working with AWS Batch on Amazon EKS. Now that AWS Batch job definitions are supported in the Cloud Control API, you can use the AWSCC provider to manage all of your Batch resources.

The choice between providers affects speed of feature adoption and code generation. The AWSCC provider offers automatic coverage as AWS publishes Cloud Control API schemas, reducing lag for new Batch capabilities. The original provider offers a mature, hand-tuned developer experience with direct SDK calls.

Core AWS Batch Architecture As Code

AWS Batch takes the heavy lifting out of running batch computing jobs. Instead of managing your own cluster of EC2 instances or containers, you define what compute you need and AWS Batch handles provisioning, scaling, and scheduling. With Terraform, you can set up the entire pipeline - compute environments, job queues, and job definitions - as code.

This guide covers everything from basic Fargate-based compute environments to advanced EC2 setups with spot instances and GPU support.

Architecture Overview

AWS Batch has three main components:

• Compute Environment: The pool of compute resources (EC2 instances or Fargate tasks)
• Job Queue: Where submitted jobs wait for compute capacity
• Job Definition: A template for your batch jobs (like a Docker Compose for batch)

IAM Roles

AWS Batch needs several IAM roles.

The three components map directly to Terraform resources. A compute environment declares the pool type, instance families, vCPU limits, and networking. A job queue binds a compute environment and defines priority and scheduling. A job definition describes container image, command, resource requirements, and environment variables.

Compute Environment Module Design

A Terraform module provisioning a Batch Compute Environment in AWS to support scalable, cost-efficient batch processing workloads is designed for use cases like data analytics, model training, ETL, and other asynchronous jobs requiring managed compute infrastructure.

Key Features

• Creates a managed compute environment for AWS Batch
• Supports custom instance types, vCPU limits, and allocation strategies
• Allows subnet and security group configuration for full VPC integration

Use Cases

• High-performance computing (HPC) jobs and render farms
• Machine learning model training and batch inference
• ETL pipelines and large-scale data preprocessing
• Genomics, financial simulations, or scientific computation

Input Variables

Outputs

The module emphasizes explicit VPC placement and tagging. Subnets and security group IDs provide full VPC integration. The environment tag supports dev/prod segregation.

Scheduled Job Infrastructure with Terraform

A walkthrough for building scheduled AWS Batch job infrastructure using Terraform focuses on the Terraform infrastructure and modules definition, and how they work together to build the entire workflow. It doesn't cover the advanced features that AWS Batch provided, and only use Terraform aws official provider and basic Terraform features.

Prerequisites documented for the demo include:

• AWS CLI (V2) is installed on the local machine
• AWS credentials setup. We use the credentials to deploy Terraform resources to the target AWS Account. In this demo, I use the same AWS profile for S3 remote backend configuration and Terraform apply
• Terraform CLI (1.3.4) is installed in the local machine. You can loosen the Terraform version restriction in versions.tf to use other close versions, however I'm not sure if the code works as expected without verification
• I'm working on Mac (MacOS Monterey) with Apple M2 Chip. So the aws provider installed is .terraform/providers/registry.terraform.io/hashicorp/aws/5.0.0/darwinarm64/terraform-provider-awsv5.0.0_x5. If you are in other operating system, you should remove .terraform.lock.hcl file from source code, and let Terraform CLI install the well-matched aws provider version according to your OS
• In the demo source code, I use default VPC, subnets and security group for EC2 instances

EC2 instance is shutdown automatically by AWS Batch.

For well-architecting and organizing Terraform structure, modules are defined for each resource group. Modules is a key feature in Terraform which helps users manage their own resources efficiently.

Batch Module: Defines resources in Batch service, including computing environment, job queue, job definition.

EventBridge Module: Defines resources in EventBridge, including event rule, rule target. One named submitbatchjobevent is used to submit Batch job as scheduled, another named capturefailedbatchevent is used to send out an alert email if a Batch job is failed.

IAM Module: Defines IAM resources, including roles, policies, instance profile. These roles are used by AWS Batch resources and EventBridge rules.

SecretManager Module: Defines secret token that may be used in Job container. It's not required in the demo project, but for your reference.

SNS Module: Defines resource in SNS, including topic and subscription.

A Makefile and shell script simplify the apply/destroy process in one command. You can find the shell script from /scripts/apply.sh in demo code.

• apply Terraform infrastructure
  make apply
• destroy Terraform resources
  make destroy

After applying Terraform resources successfully using make apply, you can submit a Batch job for testing the entire workflow.

AWS Batch on Amazon EKS with Terraform

When we announced AWS Batch support for Amazon Elastic Kubernetes Service (Amazon EKS), we laid out our thinking. Compute intensive, high scale batch workloads have materially different operational challenges from microservices. The operational overhead of these spiky and transient workloads is undifferentiated heavy lifting. Since that announcement, performance for placing pods on the cluster improved, private endpoints for tighter security were enabled, multiple containers for pods were added, and the ability for gang scheduling jobs across nodes was added.

Because AWS Batch relies on customers to provide their own clusters to scale nodes and run pods, there are a number of different types of resources to coordinate. These resources not only include the EKS cluster, Kubernetes roles and permissions on that cluster, and IAM roles for service accounts. It also encompasses AWS Batch resources, such as job queues, compute environments, and job definitions.

While the AWS Batch User Guide and documentation show how to set up all of these resources, they (by necessity) describe the process using manual steps. However, manual steps are error-prone. They are fine for a small proof of concept or tutorial, but they are not what you want to rely on for production deployments.

Terraform codifies the EKS cluster, IAM roles for service accounts, node scaling, and the Batch job queue, compute environment, and job definition into a single declarative plan. This removes manual drift and enables repeatable production deployments.

Community Terraform Module for Batch

Terraform module which creates AWS Batch resources.

See examples directory for working examples to reference.

A representative usage shows compute environments with EC2 and Spot configurations.

hcl module "batch" { source = "terraform-aws-modules/batch/aws" compute_environments = { a_ec2 = { name_prefix = "ec2" compute_resources = { type = "EC2" min_vcpus = 4 max_vcpus = 16 desired_vcpus = 4 instance_types = ["m5.large", "r5.large"] security_group_ids = ["sg-f1d03a88"] subnets = ["subnet-30ef7b3c", "subnet-1ecda77b", "subnet-ca09ddbc"] tags = { Name = "example" Type = "Ec2" } } } b_ec2_spot = { name_prefix = "ec2_spot" compute_resources = { type = "SPOT" allocation_strategy = "SPOT_CAPACITY_OPTIMIZED" bid_percentage = 20 min_vcpus = 4 max_vcpus = 16 desired_vcpus = 4 instance_types = ["m4.large", "m3.large", "r4.large", "r3.large"] security_group_ids = ["sg-f1d03a88"] subnets = ["subnet-30ef7b3c", "subnet-1ecda77b", "subnet-ca09ddbc"] } } } }

Note - any tag changes here will force compute environment replacement which can lead to job queue conflicts. Only specify tags that will be static for the lifetime of the compute environment.

The module supports allocationstrategy and bidpercentage for Spot, and explicit minvcpus, maxvcpus, desired_vcpus for capacity control. Instance type lists allow mixed families.

Operational Considerations

Compute environment replacement behavior is critical. Tag changes on compute_resources can force replacement which can lead to job queue conflicts. Only specify tags that will be static for the lifetime of the compute environment.

VPC integration requires subnet and security group IDs for the compute environment. For EKS-backed Batch, additional networking for private endpoints and cluster security groups applies.

IAM is required across Batch, EventBridge, and EKS. Roles for Batch jobs, instance profiles for EC2 compute, and service account IAM roles for EKS must be defined in Terraform to avoid manual steps.

Scheduling can be added via EventBridge rules that target Batch job submission. Failure handling can be added via EventBridge rules that trigger SNS notifications when jobs fail.

Conclusion

Using Terraform for AWS Batch delivers repeatable, auditable infrastructure for compute environments, job queues, and job definitions across EC2, Fargate, and EKS. The original Terraform AWS provider provides a hand-coded, community-driven experience with mature Batch support, while the Terraform AWS Cloud Control provider offers automatically generated coverage via the Cloud Control API, now including Batch job definitions.

Module design patterns for compute environments expose region, instance types, vCPU limits, subnets, security groups, and environment tags as inputs, and surface the environment name, ARN, and vCPU limits as outputs. Scheduled workflows are organized into Batch, EventBridge, IAM, SecretManager, and SNS modules, with Makefile automation for apply and destroy.

For EKS integration, Terraform codifies the cluster, Kubernetes RBAC, IAM roles for service accounts, and Batch job queues and compute environments, eliminating error-prone manual steps for spiky, transient HPC workloads. Community modules provide concrete examples for EC2 and Spot compute environments with allocation strategies, bid percentages, and explicit capacity controls, with the important caveat that tag changes can force compute environment replacement and cause job queue conflicts.

Sources

1. Using the Terraform AWS Cloud Control provider for managing AWS Batch resources
2. AWS Batch Compute Environment
3. Build Scheduled AWS Batch Job Infrastructure Using Terraform
4. Use Terraform to deploy a complete AWS Batch environment on Amazon EKS
5. Create Batch Compute Environments with Terraform
6. Terraform Foundation Terraform AWS Batch