Modern infrastructure as code pipelines rely heavily on Terraform to provision and manage cloud resources. While the official HashiCorp Kubernetes provider offers robust, strongly typed resources for standard objects such as Deployments, Services, ConfigMaps, and Secrets, it inherently lacks support for the vast ecosystem of extensibility mechanisms within Kubernetes. Custom Resource Definitions (CRDs), operator-managed resources, and complex multi-document manifests often fall outside the scope of the standard provider. For these scenarios, the gavinbunney/kubectl provider serves as the definitive solution, enabling users to apply arbitrary YAML manifests directly through Terraform. This approach mirrors the native kubectl apply -f command, allowing teams to manage the full lifecycle of Kubernetes resources—including drift detection, updates, and deletion—while retaining the version control, state management, and declarative power of Terraform.
Architecture and Installation of the kubectl Provider
The gavinbunney/kubectl provider is designed to be the primary tool for managing Kubernetes resources within Terraform when standard providers fall short. Its core philosophy revolves around leveraging the flexibility of YAML, the native format for Kubernetes configuration. The provider processes free-form YAML, applies it to the cluster, and tracks the object in Terraform state. This tracking mechanism is critical, as it enables seamless handling of creation, updates, and deletions, including the detection of drift between the desired state defined in code and the actual state of the cluster.
Installing the provider is a standard procedure within Terraform's module initialization workflow. The provider can be installed and managed automatically by Terraform, removing the need for manual binary placement in most modern setups. To integrate the provider into an existing Terraform configuration, specific version constraints must be defined to ensure compatibility with the Terraform core and other required providers. A typical versions.tf file for a hybrid setup utilizing both the official Kubernetes provider and the kubectl provider looks as follows:
```hcl
terraform {
required_version = ">= 1.5.0"
required_providers {
kubectl = {
source = "gavinbunney/kubectl"
version = "~> 1.14"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.25"
}
}
}
```
Configuration of the provider requires pointing it to the correct cluster context. The kubectl provider utilizes the same kubeconfig mechanism as the official Kubernetes provider, which simplifies credential management. By configuring both providers to point at the same cluster, teams can maintain a unified access layer. The provider block specifies the path to the kubeconfig file and the context to use:
hcl
provider "kubectl" {
config_path = "~/.kube/config"
config_context = "my-cluster"
}
This configuration ensures that Terraform authenticates against the intended cluster using the existing Kubernetes service account or API keys stored in the kubeconfig. The config_context argument is particularly useful in environments with multiple clusters defined in a single kubeconfig file, allowing precise targeting without modifying the underlying Kubernetes configuration files.
Applying Single-Document YAML Manifests
The fundamental resource within the kubectl provider is kubectl_manifest. This resource accepts a YAML string and applies the resource to the cluster, effectively executing a kubectl apply with the provided content. Unlike dynamic resources that require specific provider support for every attribute, kubectl_manifest accepts the raw YAML definition, making it ideal for resources that the official provider does not support natively.
A basic example of applying a single resource is the creation of a Namespace with specific labels. The following configuration demonstrates how to define a custom namespace labeled for Terraform management:
hcl
resource "kubectl_manifest" "namespace" {
yaml_body = <<YAML
apiVersion: v1
kind: Namespace
metadata:
name: custom-app
labels:
managed-by: terraform
YAML
}
When this configuration is applied, Terraform tracks the resource in its state file. Subsequent runs will compare the yaml_body against the live cluster object. If no changes are detected, the resource is marked as up to date. If changes are made to the YAML, Terraform plans an update operation. This capability is particularly powerful for Custom Resource Definitions (CRDs) and operator-managed objects, where the schema is dynamic and not known to the official provider at compile time.
Handling Multi-Document YAML Files
In enterprise Kubernetes environments, software is frequently distributed as single YAML files containing multiple resources, such as an operator bundle or a certificate manager installation. Manually splitting these files into individual Terraform resources is error-prone and difficult to maintain. The kubectl provider addresses this with the kubectl_file_documents data source, which parses a file and splits it into individual documents.
The workflow involves loading the multi-document YAML file and iterating over the resulting manifest list. The following configuration demonstrates installing the cert-manager bundle using this pattern:
```hcl
data "kubectlfiledocuments" "cert_manager" {
content = file("${path.module}/manifests/cert-manager.yaml")
}
resource "kubectlmanifest" "certmanager" {
foreach = data.kubectlfiledocuments.certmanager.manifests
yaml_body = each.value
# Wait for Deployment and APIService resources in the bundle to roll out
waitforrollout = true
}
```
The for_each argument creates a separate kubectl_manifest resource for each document in the YAML file. The yaml_body is set to the content of each individual document. A critical parameter in this context is wait_for_rollout. When set to true, the provider waits for Deployments and APIService resources within the bundle to complete their rollout before marking the resource as created. This prevents Terraform from proceeding with subsequent resources before the prerequisite workloads are fully available, ensuring a stable deployment pipeline.
This pattern is highly effective for installing CRD bundles, operator manifests, and other multi-resource YAML files. It allows teams to treat complex Kubernetes installations as single atomic units within Terraform, simplifying dependency management. Additionally, the provider supports fetching YAML directly from URLs, enabling the application of remote manifests without storing them in the local repository.
Dynamic Templating and Variable Injection
Static YAML files limit the reusability of Terraform modules. To create flexible configurations, the kubectl provider integrates with Terraform's template engine. This allows for dynamic content injection into YAML manifests based on variables, local values, or outputs from other resources. The templatefile function is commonly used to render template files with variables before passing them to the kubectl_manifest resource.
Consider a scenario where an IngressRoute needs to be created for multiple applications with varying hostnames, services, and ports. A template file can be defined with placeholders:
yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: ${name}
namespace: ${namespace}
spec:
entryPoints:
- websecure
routes:
- match: Host(`${domain}`)
kind: Rule
services:
- name: ${service}
port: ${port}
tls:
certResolver: letsencrypt
The Terraform configuration then uses templatefile to populate these placeholders:
hcl
resource "kubectl_manifest" "ingress_route" {
yaml_body = templatefile("${path.module}/manifests/ingress-route.yaml.tpl", {
name = var.app_name
namespace = var.namespace
domain = var.domain
service = var.service_name
port = var.service_port
})
}
This approach allows a single template to serve multiple environments or applications. The variables app_name, namespace, domain, service_name, and service_port can be defined in variables.tf, making the module highly reusable. The resulting YAML is generated at plan time, ensuring that the applied manifest matches the intended configuration exactly.
Server-Side Apply and Conflict Resolution
Kubernetes clusters are often managed by multiple controllers and tools simultaneously. Client-side apply, which is the default behavior in many CLI tools, can lead to conflicts when other controllers modify the same resources. The kubectl provider supports server-side apply, which delegates the merging logic to the Kubernetes API server. This method is superior at handling conflicts with other controllers because it uses the field management mechanism provided by the API.
To enable server-side apply, the server_side_apply attribute is set to true in the resource definition. This is particularly important for resources that are frequently updated by operators or other automation systems:
hcl
resource "kubectl_manifest" "deployment" {
yaml_body = <<YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: production
spec:
replicas: 3
YAML
server_side_apply = true
}
By using server-side apply, Terraform asserts its ownership over the fields it manages, while allowing other controllers to manage their respective fields without triggering unnecessary update loops or conflicts. This feature is critical for maintaining stable production environments where multiple sources of configuration coexist.
Integration with the Official Kubernetes Provider
While the kubectl provider is versatile, it is often used in conjunction with the official HashiCorp Kubernetes provider. The official provider uses the kubernetes_manifest resource for similar purposes but operates with a different data model. In the official provider, the resource has two primary attributes: manifest and object.
The manifest attribute represents the desired configuration specified by the user. The object attribute contains the end state returned by the Kubernetes API server after the resource is created. It is a common pitfall to reference the manifest attribute in outputs or dependencies, as it does not reflect the actual state of the cluster. Instead, the object attribute should be used, as it includes all fields added by the API server, such as UID, resource version, and default values.
The object attribute contains significantly more fields than the user-specified manifest because Terraform generates a schema containing all possible resource attributes that the Kubernetes API server could add. For example, when creating a custom resource based on a CRD, the object attribute will include metadata fields such as uid, resourceVersion, and selfLink that are not present in the input manifest.
Consider the following example from the official provider's documentation, where a custom resource is created:
hcl
resource "kubernetes_manifest" "my_new_crontab" {
manifest = {
"apiVersion" = "stable.example.com/v1"
"kind" = "CronTab"
"metadata" = {
"name" = "my-new-cron-object"
"namespace" = "default"
}
"spec" = {
"cronSpec" = "* * * * */5"
"image" = "my-awesome-cron-image"
}
}
}
When this resource is applied, the plan indicates that both manifest and object will be set. The object attribute will mirror the manifest but will be enriched with server-generated fields. This distinction is crucial for downstream dependencies. If another resource requires the resourceVersion or uid of the created object, it must reference kubernetes_manifest.my_new_crontab.object.uid rather than the manifest attribute.
Drift Detection and State Management
One of the most significant advantages of managing Kubernetes resources in Terraform, whether via the official provider or the kubectl provider, is the automatic drift detection. Kubernetes resources are often modified directly via kubectl edit or by other tools, causing the actual state to diverge from the desired state defined in Terraform code.
Both providers detect this drift during the terraform plan phase. If a resource in the cluster has been modified outside of Terraform, the plan will show an update operation to reconcile the difference. For the kubectl_manifest resource, this comparison is based on the YAML content. If the live YAML differs from the yaml_body attribute, Terraform plans an update. This ensures that the infrastructure as code definition remains the single source of truth.
The provider handles creation, updates, and deletions seamlessly. When a resource is destroyed, the provider issues a delete request to the Kubernetes API. This is particularly useful for ephemeral environments or CI/CD pipelines where resources need to be torn down cleanly after use. The state file retains the history of these operations, allowing for rollback and audit trails.
Verification and CLI Interoperability
After applying configurations via Terraform, it is best practice to verify the state of the resources using the kubectl command-line tool. This interoperability ensures that the resources are visible and functional within the cluster. For example, after creating a CRD using Terraform, the existence of the CRD can be confirmed with:
bash
kubectl get crds crontabs.stable.example.com
If the CRD was created successfully, the command will list the CRD name and creation timestamp. If the CRD did not exist or failed to apply, the command would return an error indicating that the server does not have the resource type.
Similarly, for custom resources, verification can be performed by listing the resources in the relevant namespace:
bash
kubectl get crontabs
This command will list any CronTab resources in the default namespace. If no resources are found, it will return "No resources found in default namespace." This step is critical for debugging issues where Terraform reports success but the resources are not behaving as expected in the cluster.
Advanced Use Cases and Best Practices
The kubectl provider is widely used in large Kubernetes installations to completely manage the lifecycle of Kubernetes resources. Its ability to handle arbitrary YAML makes it indispensable for teams that rely on third-party tools and operators that do not have dedicated Terraform providers.
When using the provider, several best practices should be observed:
- Use server-side apply for resources that are managed by multiple controllers.
- Leverage
wait_for_rolloutfor Deployments and APIServices to ensure dependencies are met. - Use
templatefilefor dynamic configurations to avoid code duplication. - Keep YAML files modular, using the
kubectl_file_documentsdata source for multi-resource bundles. - Monitor the Terraform state file to understand which resources are being managed by Terraform and which are external.
The provider's ability to process directories of YAML files and support inline templating makes it a powerful tool for complex Kubernetes environments. By integrating the kubectl provider into Terraform workflows, teams can achieve complete infrastructure as code coverage for their Kubernetes clusters, ensuring consistency, reliability, and reproducibility across all environments.
Conclusion
The integration of the gavinbunney/kubectl provider into Terraform workflows represents a critical advancement in managing modern Kubernetes infrastructure. By enabling the application of arbitrary YAML manifests, the provider bridges the gap between the rigid, typed resources of the official Kubernetes provider and the flexible, extensible nature of the Kubernetes ecosystem. It empowers engineers to manage CRDs, operators, and complex multi-document bundles within a unified Terraform state, ensuring that all infrastructure is version-controlled and drift-detected.
The detailed understanding of attributes such as yaml_body, server_side_apply, and the interaction between manifest and object attributes in related providers is essential for effective usage. The provider's support for dynamic templating, server-side apply, and multi-document processing addresses the most common pain points in Kubernetes automation. As the Kubernetes ecosystem continues to evolve with new custom resources and complex deployment patterns, the kubectl provider remains a vital component in the toolset of any DevOps team focused on reliability and automation. The seamless integration with existing kubeconfig files and its robust state management capabilities make it a standard choice for production-grade Kubernetes deployments managed by Terraform.