Mastering Kubernetes Manifest Management with Terraform

The evolution of Infrastructure as Code (IaC) has transitioned from simple virtual machine provisioning to the complex orchestration of containerized workloads. For engineers managing Kubernetes clusters, the challenge lies in bridging the gap between the desired state of the cluster and the declarative nature of Terraform. While many cloud-specific providers (such as azurerm for Azure Kubernetes Service or aws for Elastic Kubernetes Service) handle the creation of the cluster itself, managing the internal objects—namespaces, pods, services, and Custom Resource Definitions (CRDs)—requires a deeper dive into the Kubernetes provider ecosystem.

Managing Kubernetes manifests through Terraform allows teams to apply the same version control, peer review, and deployment pipelines to their application manifests as they do to their core infrastructure. This ensures that the environment is reproducible and that drift can be detected and remediated automatically.

Understanding the Terraform Kubernetes Provider Architecture

At its core, a Terraform provider is a plugin that acts as an interface between the Terraform CLI and a specific API. The Kubernetes provider converts Hashicorp Configuration Language (HCL) into API calls that the Kubernetes API server understands. This enables the direct deployment and management of objects on a K8s cluster.

When configuring the provider, Terraform must be able to communicate with the cluster. For local environments like minikube, this is typically achieved by ensuring the kubeconfig file located at ~/.kube/config contains the necessary cluster information and certificates. For cloud-managed clusters like AKS, the process involves authenticating to retrieve a token from the cluster before the provider can establish a connection.

The provider ecosystem for Kubernetes in Terraform is split between the official native provider and community-driven alternatives like the terraform-provider-kubectl, each serving distinct needs based on whether the user prefers HCL-native definitions or raw YAML.

Working with the Native kubernetes_manifest Resource

The kubernetes_manifest resource is the primary mechanism within the official provider for managing Kubernetes objects, particularly those defined by Custom Resource Definitions (CRDs). Unlike standard resources (like kubernetes_deployment), the kubernetes_manifest allows for a more flexible definition of the object state.

Deploying Custom Resources

When deploying a resource that is not natively known to the provider—such as a function in an OpenFaaS environment—the kubernetes_manifest resource is essential. Consider a scenario where you are deploying a "showcow" function using the OpenFaaS CRD. The manifest for such a resource would traditionally look like this in YAML:

yaml apiVersion: openfaas.com/v1 kind: Function metadata: name: showcow namespace: openfaas-fn spec: name: showcow handler: node show_cow.js image: alexellis2/ascii-cows-openfaas:0.1

To use this with the official Terraform Kubernetes provider, the YAML must be converted to HCL. This can be accomplished using the yamldecode() function in combination with the terraform console command:

bash $ echo 'yamldecode(file("cows.yaml"))' | terraform console

The resulting HCL output would then be wrapped in a kubernetes_manifest resource:

hcl resource "kubernetes_manifest" "openfaas_fn_showcow" { manifest = { apiVersion = "openfaas.com/v1" kind = "Function" metadata = { name = "showcow" namespace = "openfaas-fn" } spec = { name = "showcow" handler = "node show_cow.js" image = "alexellis2/ascii-cows-openfaas:0.1" } } }

Lifecycle Management and Resource Actions

Terraform tracks the state of these manifests meticulously. During a terraform plan or apply, the CLI uses specific symbols to indicate the intended action:

+ : Create. This indicates the resource does not exist in the state and will be added to the cluster.
- : Destroy. This indicates the resource exists in the state but is no longer in the configuration and will be removed.

For example, when creating a new CronTab object (stable.example.com/v1), Terraform will display the kubernetes_manifest.my_new_crontab resource with the + symbol, detailing the specific API version, kind, and spec (such as the cronSpec and image) being applied.

Conversely, when a resource is marked for destruction, Terraform provides a detailed diff. If a user deletes a function from their configuration, the plan will show the kubernetes_manifest.openfaas_fn_showcow being destroyed, including the removal of labels (e.g., com.openfaas.scale.max = "6") and limits (e.g., cpu = "100m", memory = "64Mi"). It is critical to note that destruction is irreversible ("There is no undo").

The terraform-provider-kubectl Alternative

While the official provider requires conversion to HCL, the terraform-provider-kubectl is designed specifically for users who prefer to keep their manifests in YAML format. This provider is often favored in large-scale Kubernetes installations because it allows "free-form" YAML to be processed and applied directly against the cluster.

The kubectl_manifest Resource

The core of this provider is the kubectl_manifest resource. It eliminates the need for yamldecode() by allowing the user to pass a YAML string directly into the yaml_body attribute.

hcl resource "kubectl_manifest" "namespace_with_labels" { yaml_body = <<-YAML apiVersion: v1 kind: Namespace metadata: name: production labels: environment: production managed-by: terraform YAML server_side_apply = true force_conflicts = false }

Advanced Configuration: Server-Side Apply and Conflict Management

One of the most powerful features of the kubectl_manifest resource is the support for Server-Side Apply (SSA). When server_side_apply = true, the provider utilizes kubectl as the field manager.

The force_conflicts attribute provides a safety mechanism for multi-controller environments:

force_conflicts = false: Terraform will fail if another controller (such as an operator or a manual kubectl edit) owns the fields that Terraform is attempting to modify. This prevents accidental overwrites.
force_conflicts = true: Terraform will forcefully take ownership of the fields, overwriting existing values.

Comparison of Kubernetes Management Approaches

The following table summarizes the differences between using the official Kubernetes provider and the terraform-provider-kubectl.

Feature	Official Kubernetes Provider	terraform-provider-kubectl
Input Format	HCL (converted from YAML)	Native YAML
Primary Resource	`kubernetes_manifest`	`kubectl_manifest`
Configuration Logic	Uses `yamldecode()` for external files	Uses `yaml_body` heredocs or files
Conflict Handling	Standard Terraform State	`force_conflicts` & Server-Side Apply
Drift Detection	Native state tracking	Seamless YAML tracking
Ease of Transition	Requires HCL translation	Direct copy-paste of K8s manifests

Implementation Strategies and Best Practices

Successfully managing Kubernetes manifests requires a strategic approach to directory structure and dependency management.

Directory Layout

For local development with tools like minikube, it is recommended to maintain a clean directory structure. A typical setup includes:
- provider.tf: Defines the Kubernetes provider and its version.
- main.tf: Contains the core resource definitions.
- manifests/: A directory containing raw YAML files for the application.

Handling Resource Dependencies

When deploying complex stacks—particularly those involving operators and their corresponding Custom Resources—the order of operations is vital. You cannot deploy a Function resource if the OpenFaaS operator has not yet established the Function CRD in the cluster.

In Terraform, this is managed through:
- depends_on: Explicitly telling Terraform to wait for the operator deployment to complete before initiating the manifest deployment.
- Implicit Dependencies: Referencing the output of one resource in another.

Converting YAML to HCL Workflow

For teams committed to the official provider, the workflow for converting existing manifests is standardized:

Define the YAML manifest (e.g., cows.yaml).
Use terraform console to validate the yamldecode(file("cows.yaml")) output.
Map the resulting object into a kubernetes_manifest resource.
Run terraform init to initialize the working directory. Note that if modules or backend configurations change, terraform init must be rerun.

Technical Specification Analysis: Resource Attributes

Understanding the specific attributes of the manifest resources is key to avoiding deployment failures. Whether using the official or the kubectl provider, several common fields are present across almost all Kubernetes manifest implementations in Terraform.

Metadata and Spec Structures

The metadata block is used for cluster-level identification. This includes:
- name: The unique identifier for the resource.
- namespace: The logical partition within the cluster (e.g., default or openfaas-fn).
- labels: Key-value pairs used for organization and selection (e.g., environment: production).

The spec block contains the desired state of the resource. For a functional resource like a containerized function, this includes:
- image: The container image registry path (e.g., alexellis2/ascii-cows-openfaas:0.1).
- handler: The command or script used to execute the function (e.g., node show_cow.js).
- limits: Hardware constraints for the pod, such as cpu = "100m" and memory = "64Mi".

Conclusion

Integrating Kubernetes manifests into Terraform transforms cluster management from a series of imperative kubectl apply commands into a disciplined, declarative pipeline. For engineers who prioritize tight integration with the Hashicorp ecosystem and prefer HCL, the official kubernetes_manifest resource provides a robust way to handle standard and custom resources, though it requires a conversion step from YAML. For those managing massive scales of YAML or working in environments where multiple controllers manage the same resources, the terraform-provider-kubectl offers superior flexibility through native YAML support and Server-Side Apply capabilities.

The choice between these providers depends on the team's existing workflow. If the organization already has a library of validated YAML manifests, the kubectl_manifest resource reduces friction and minimizes the risk of conversion errors. However, if the goal is to treat Kubernetes objects as first-class Terraform citizens with full HCL power, the official provider is the logical choice. Regardless of the tool, the ability to detect drift and manage the lifecycle—from creation to the final destroy action—ensures that the Kubernetes cluster remains stable, secure, and reproducible.