Terraform and Apache Airflow are both foundational tools for modern data engineering. Terraform builds the infrastructure those workflows run on while Airflow orchestrates the DAGs that move and transform data. When the two are wired together deliberately, infrastructure provisioning becomes repeatable, auditable, and governed by the same principles as the pipelines themselves. The integration patterns range from using official managed modules for AWS MWAA and Google Cloud Composer to custom Terraform configurations that generate DAG files, trigger refreshes, and provision self-managed Airflow stacks on EC2, RDS, and Elasticache.
Managed Workflows for Apache Airflow with Terraform
AWS introduced a Terraform module for Managed Workflows for Apache Airflow, MWAA. Terraform is an open-source infrastructure as code software tool created by HashiCorp. Even with familiarity with Terraform and featuring modules in a blog, hands-on use with the MWAA module represents a new application of the tool.
The AWS-IA team put together a new Managed Workflows for Apache Airflow module for Terraform. Time was spent playing around with this, testing it and adding tweaks to the docs. The module provides a declarative way to configure MWAA environments without manual console steps.
Integrating Airflow Pipelines with Terraform Provisioning
Your pipeline’s done, your infrastructure’s built, and your team still spends half its day sorting out who can trigger what. Sound familiar? Airflow and Terraform are brilliant apart, but together they can either feel like magic or chaos depending on how you wire them. Getting Airflow Terraform right means making them talk cleanly—no dangling permissions, no manual state edits, no security gray zones.
Airflow runs DAGs. Terraform builds everything those DAGs depend on. When integrated, Terraform defines the infrastructure while Airflow automates the workflows that use it. Instead of clicking your way through IAM or refreshing a credentials file at midnight, Airflow can trigger Terraform runs using well-defined variables and identity-aware automation. The result is infrastructure provisioning that feels less like a ritual and more like a system.
Here’s the logic: Airflow passes execution context to Terraform through environment variables or cloud backends. Terraform then applies state and updates resources. The Airflow scheduler manages access through service accounts or federated identity, ideally using OIDC from providers like Okta or AWS IAM. This keeps jobs secure and auditable, because every task runs under a predictable identity with enforced permissions. No mystery users, no shared keys.
To troubleshoot this setup, start with identity mapping. If Terraform runs with more privilege than Airflow requires, restrict it through role-based access control. Rotate secrets regularly, or better yet, remove them entirely by using short-lived tokens. For error handling, keep Terraform state remote and versioned, so Airflow retries don’t collide with manual updates. It’s not glamorous work, but it keeps your environment sane.
Benefits of binding Airflow with Terraform include consistent provisioning, auditable changes, and removal of manual credential handling.
Automating DAG Deployment with Terraform Orchestration
Accelerating Airflow DAG Deployment through Automated Terraform Orchestration is the focus of a project that leverages Terraform, an infrastructure as code tool, to automate the deployment of custom Airflow DAGs. The goal is to simplify deployment, reduce errors, and ensure consistency across environments.
The project comprises a Terraform configuration that automates the creation of local files from provided DAG templates and triggers Airflow DAG refresh to seamlessly integrate the new DAGs. By abstracting manual steps and providing a standardized deployment approach, this solution enhances deployment efficiency and reduces the risk of errors.
How to Use
- Clone the GitHub Repository:
Clone the repository to your local machine using the following command:
git clone https://github.com/Zymberg/terraform-atomization
Install Dependencies:
If you don’t have Airflow and Terraform installed, follow the instructions provided in the README file of the repository.Navigate to Airflow Directory:
Inside the cloned repository, navigate to the airflow directory. You'll find two folders:
dags
: New DAGs will be placed here after they are created.
templatedags
: Contains template DAGs (welcomedag.py and goodbye_dag.py).Update Variables:
Open the terraform/variable.tfvars file and add the following variables:
new dags file names, new dag ids, template dags file names.
For testing purposes we’ve added the dags below:
dag_filenames = [
"welcome_john_dag.py",
"goodbye_john_dag.py"
]
dag_ids = [
"welcome_john",
"goodbye_john"
]
template_filenames = [
"welcome_dag",
"goodbye_dag"
]
- Build Docker Image:
In your terminal, navigate to the project directory and build the Docker image using:
docker build -t my-airflow .
- Run Docker Compose:
Start the Airflow container by running Docker Compose:
docker-compose up -d
- Initialize Terraform:
Navigate to the terraform directory and initialize Terraform:
terraform init
- Deploy DAGs:
Apply the Terraform configuration to create and deploy the new DAGs:
terraform apply -var-file="variable.tfvars"
- Approve Deployment:
When prompted, enter yes to approve the deployment:
Do you want to perform these actions? Terraform will perform the actions described above. Only ‘yes’ will be accepted to approve. Enter a value: yes and press enter.
The workflow shows how Terraform can be used not only for infrastructure but for generating and delivering DAG artifacts in a repeatable manner.
Provisioning Self-Managed Airflow on AWS with Terraform
Easily provision resources on AWS for a highly scalable and secure airflow environment.
The reference architecture uses Terraform to provision resources in a VPC.
- Airflow's services are deployed on an EC2 instance.
- The Airflow metadata database is deployed on RDS (Postgres).
- The Airflow message broker is deployed on Elastic Cache (Redis).
- AWS Secrets Manager is used to securely feed environment variables such as airflow's fernet key into the EC2 instance.
Prerequisites are an AWS Account and Terraform.
Create an EC2 key pair or use an existing one:
aws ec2 create-key-pair --key-name MyKeyPair
First enter the Terraform directory:
cd terraform
- See a plan of resources
terraform plan
- Provision resources
terraform apply
- Destroy resources
terraform destroy
The table below summarizes the AWS components provisioned by Terraform in this self-managed pattern.
| Component | AWS Service | Role in Airflow |
|---|---|---|
| Compute | EC2 instance | Hosts Airflow webserver, scheduler, workers |
| Metadata store | RDS Postgres | Airflow metadata database |
| Message broker | Elasticache Redis | Task queue and DAG state |
| Secrets | AWS Secrets Manager | Supplies fernet key and env vars to EC2 |
| Network | VPC | Isolates Airflow resources |
Creating Managed Airflow Environments on Google Cloud with Terraform
This page is a companion to the main page about creating environments. It demonstrates how to set up a Managed Airflow environment and a user-managed service account for this environment in an existing Google Cloud project with Terraform. You can use this page as a start, then add more configuration parameters for your environment, as needed.
About Terraform modules for Managed Airflow
There are other Terraform modules maintained by the open source community. To resolve problems with such modules, you can create issues in their repositories on GitHub. One example of an open source project maintained on GitHub is the terraform-google-composer module, which is a part of Terraform blueprints and modules for Google Cloud. While Google contributes to this project, the module is not maintained by Google and is not related to the Terraform provider for Google Cloud.
Before you begin
This guide assumes that you have a Google Cloud project with configured billing.
Authenticate with Google Cloud
To authenticate with Google Cloud, run:
gcloud auth application-default login
For more information about this command, see gcloud auth application-default.
Configure the Google provider in Terraform
Specify your existing project ID and a default region for resources. Your Managed Airflow environment uses this region.
The google-beta provider supports Managed Airflow features that are in Preview.
The Google Cloud pattern emphasizes using Terraform to define the Composer environment and a dedicated service account, keeping environment creation declarative and repeatable.
Conclusion
Terraform and Apache Airflow integration is not about replacing one tool with the other. It is about drawing a clean boundary between what Terraform builds and what Airflow runs. Terraform defines infrastructure, service accounts, storage, and networking as code. Airflow consumes that infrastructure and orchestrates workflows against it, optionally triggering further Terraform changes through identity-aware automation.
Managed services like AWS MWAA and Google Cloud Composer reduce operational burden, and official or community Terraform modules make those environments reproducible. Self-managed patterns on AWS show how Terraform can stand up the full stack: EC2 for Airflow, RDS Postgres for metadata, Elasticache Redis for messaging, and Secrets Manager for secure configuration.
Automated DAG deployment via Terraform demonstrates a second axis of integration where Terraform generates or delivers DAG artifacts from templates and refreshes Airflow to pick them up. This reduces manual copy-paste errors and enforces naming conventions across environments.
Security and operational hygiene remain central. Identity mapping, role-based access control, short-lived tokens via OIDC, remote versioned Terraform state, and secretless patterns keep the combined system auditable. When these practices are followed, Airflow and Terraform move from feeling like magic or chaos to a predictable system where infrastructure changes and workflow changes are both tracked, reviewed, and applied consistently.