Terraform AWS EKS Add-on Management for Production Kubernetes

Infrastructure as Code for Amazon EKS requires precise handling of AWS-managed Kubernetes components. The aws_eks_addon resource provides declarative control over add-ons installed directly through the EKS API, distinct from self-managed Helm charts or manifests. Terraform enables version pinning, reproducible provisioning, and automated updates for critical cluster components such as VPC CNI, CoreDNS, kube-proxy, and CSI drivers.

Why Manage EKS Add-ons with Terraform

Terraform is an open-source tool for defining and provisioning infrastructure using a straightforward configuration language. It helps you manage infrastructure as code, which makes it easier to track changes and collaborate within teams.

Using Terraform with AWS EKS offers several advantages:

  • Consistency: Infrastructure can be provisioned consistently across multiple environments, which is especially useful when scaling from development to production.
  • Version Control: Changes can be tracked and managed within version control systems, such as Git. For instance, you can review and roll back changes if necessary.
  • Automation: Terraform automates the entire provisioning process. This automation helps reduce human error, lowering the chances of misconfigurations that could lead to downtime.
  • Modularity: You can create reusable components with Terraform modules, simplifying the management of complex infrastructures

You can leverage Kubernetes's power while enjoying AWS's top-notch security, scalability, and reliability features.

Add-on Resource Mapping and Timing

Add-ons are provisioned via the EKS API and managed by AWS, ensuring compatibility with the cluster version.

The module creates two separate aws_eks_addon resources to handle different timing requirements. Add-ons are provisioned via the EKS API and managed by AWS, ensuring compatibility with the cluster version.

Typical Add-on Timing Pattern

Add-on Timing Rationale
vpc-cni Before Compute Nodes need CNI to obtain IP addresses
pod-identity-agent Before Compute Required for Pod Identity authentication
coredns After Compute Needs nodes to schedule pods
kube-proxy After Compute Needs nodes for DaemonSet deployment
aws-ebs-csi-driver After Compute Storage driver for workloads
aws-efs-csi-driver After Compute Storage driver for workloads

The var.addons input variable accepts a map where keys are addon identifiers and values are configuration objects.

Each addon configuration supports the following attributes:

Add-on Configuration Attributes

Version Resolution Logic: If addon_version is not specified, the module queries data.aws_eks_addon_version.this with most_recent flag default true to automatically select the latest compatible version for the cluster's Kubernetes version.

The before_compute flag controls which Terraform resource is used and therefore which dependencies are enforced. This is critical for add-ons like VPC-CNI that must be functional before nodes attempt to join the cluster.

Resource Selection Based on before_compute Flag

Dependency Resolution Table

Resource Dependencies Terraform Expression
aws_eks_addon.before_compute EKS cluster only No explicit depends_on block
aws_eks_addon.this Cluster + all compute dependson = [module.fargateprofile, module.eksmanagednodegroup, module.selfmanagednodegroup]

Critical Before-Compute Add-ons: vpc-cni and pod-identity-agent should typically use before_compute = true because nodes require these for network configuration and Pod Identity authentication during bootstrap.

Add-ons that require AWS API access e.g., CSI drivers, AWS Load Balancer Controller need IAM credentials

Declarative Add-on Definitions

For IAM roles and service account authentication IRSA/Pod Identity, see page 2.2. For Karpenter-specific addon infrastructure, see page 4.1.

The aws_eks_addon resource allows us to specify each addon we want to install on our cluster. We utilize a for_each loop to iterate over a list of addons defined in a variable, creating an addon configuration for each:

hcl resource "aws_eks_addon" "addons" { for_each = { for addon in var.addons : addon.name => addon } cluster_name = var.cluster_name addon_name = each.value.name addon_version = each.value.version resolve_conflicts_on_update = "OVERWRITE" }

variables.tf

```hcl
variable "aws_region" {
type = string
description = "The AWS region for the provider to deploy resources into."
}

variable "cluster_name" {
type = string
description = "The name of the EKS cluster."
}

variable "addons" {
type = list(object({
name = string
version = string
}))
default = [
{
name = "kube-proxy"
version = "v1.27.1-eksbuild.1"
},
{
name = "vpc-cni"
version = "v1.12.6-eksbuild.2"
},
{
name = "coredns"
version = "v1.10.1-eksbuild.1"
},
{
name = "aws-ebs-csi-driver"
version = "v1.25.0-eksbuild.1"
}
]
}
```

Provider Configuration

hcl provider "aws" { region = "us-west-2" # Change to your desired region

Version Pinning and Cluster Configuration

Pinning add-on versions is a production best practice.

Example version variables:

```hcl
awsekscluster.main.version
oidcprovider = awsiamopenidconnect_provider.eks

Add-on versions

vpccniversion = "v1.15.4-eksbuild.1"
corednsversion = "v1.10.1-eksbuild.6"
kube
proxyversion = "v1.28.2-eksbuild.2"
ebs
csi_version = "v1.26.0-eksbuild.1"
```

VPC CNI configuration

hcl vpc_cni_config = { enable_prefix_delegation = true enable_pod_eni = true enable_network_policy = true warm_ip_target = 5 minimum_ip_target = 10 }

CoreDNS configuration

hcl coredns_config = { replica_count = 3 resources = { limits = { cpu = "200m" memory = "256Mi" } requests = { cpu = "100m" memory = "128Mi" } } }

Dependency enforcement

hcl depends_on = [ aws_eks_node_group.main ]

Deploying Infrastructure

Within this directory, create a file named main.tf. This file will contain the configuration for your EKS cluster.

With your Terraform configuration defined, you’re now ready to deploy your EKS cluster and addons. Follow these steps:

  • Initialize Terraform: Run terraform init to initialize your Terraform workspace. This command will download the necessary Terraform providers.
  • Plan your deployment: Execute terraform plan to review the actions Terraform will perform

After adding the add-ons to your configuration, run the following command again:

bash terraform apply

This will deploy the specified add-ons to your EKS cluster.

Verifying Add-on Health

Verifying Your EKS Cluster and Add-ons

To verify that your EKS cluster and add-ons are running correctly, use these kubectl commands:

bash kubectl get nodes kubectl get pods -n kube-system

These commands show the status of your nodes and the pods running in the kube-system namespace, where most add-ons are deployed.

Verify add-on functionality:

```bash

Check add-on status

terraform show | grep addon

Verify with AWS CLI

aws eks list-addons --cluster-name my-cluster
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni

Test VPC CNI

kubectl get pods -n kube-system -l k8s-app=aws-node
kubectl logs -n kube-system -l k8s-app=aws-node

Test CoreDNS

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl run test-dns --image=busybox:1.28 --rm -it --restart=Never -- nslookup kubernetes.default

Test kube-proxy

kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl get ds -n kube-system kube-proxy
```

Updating Add-ons Safely

Update add-on versions:

```hcl

Update version in terraform

resource "awseksaddon" "vpccni" {
addon
version = "v1.16.0-eksbuild.1" # New version
# ...
}
```

Apply changes

terraform plan

terraform apply

Check compatibility:

```bash

List available versions

aws eks describe-addon-versions \
--addon-name vpc-cni \
--kubernetes-version 1.28

Check compatibility

aws eks describe-addon-configuration \
--addon-name vpc-cni \
--addon-version v1.16.0-eksbuild.1
```

Best Practices

  • Pin add-on versions for production clusters
  • Test add-on updates in non-production first
  • Use PRESERVE for conflict resolution on updates
  • Configure appropriate resource limits
  • Enable monitoring and logging for add-ons
  • Use IAM roles for service accounts IRSA for add-ons
  • Document custom configurations
  • Set up alerts for add-on health
  • Regularly review and update add-ons
  • Keep Terraform state secure

Managing EKS add-ons with Terraform provides version control, reproducibility, and automation for critical cluster components

Conclusion

Managing EKS add-ons with Terraform provides version control, reproducibility, and automation for critical cluster components. The aws_eks_addon resource abstracts AWS-managed installation while exposing version pinning, configuration objects, and dependency ordering. Before-compute add-ons like vpc-cni and pod-identity-agent must be provisioned prior to node group creation to avoid bootstrap failures, whereas after-compute add-ons like coredns and kube-proxy depend on nodes for scheduling.

Version resolution logic allows automatic selection of the latest compatible version when addon_version is omitted, but production environments require explicit pinning to prevent unintended upgrades. The before_compute flag determines resource selection and enforced dependencies, ensuring that network and identity prerequisites are satisfied before compute resources join the cluster.

Operational workflows benefit from Terraform plan and apply cycles, AWS CLI verification of addon status, and kubectl checks for pod health in kube-system. Consistent verification steps for vpc-cni, coredns, and kube-proxy provide confidence that the control plane and data plane are functioning as intended.

Long-term reliability depends on monitoring, logging, IRSA for add-on IAM access, and regular review of add-on versions against the EKS Kubernetes version. With declarative definitions, reusable modules, and controlled update processes, Terraform enables safe, repeatable EKS add-on management across development to production.

Sources

  1. EKS Add-ons CoreDNS VPC CNI Terraform
  2. Mastering AWS EKS and Add-ons Deployment with Terraform
  3. Terraform AWS EKS Add-ons and Extensions
  4. Managing EKS Add-ons with Terraform

Related Posts