Orchestrating k3s Clusters via Terraform and k3sup on Google Cloud

The intersection of Infrastructure as Code (IaC) and lightweight container orchestration has culminated in the adoption of k3s, a certified Kubernetes distribution specifically engineered for simplicity and efficiency. While traditional Kubernetes (k8s) deployments are notorious for their complexity—requiring a delicate balance of separate master nodes for the control plane, dedicated worker nodes for workloads, and independent clusters for the etcd database to ensure High Availability—k3s fundamentally disrupts this paradigm. By packaging the entire distribution into a single binary of less than 40MB, k3s reduces the overhead of dependencies and drastically minimizes the time required to move from a blank slate to a production-ready cluster. This streamlined architecture makes it an ideal candidate for Edge computing, Internet of Things (IoT) use cases, and rapid development environments where the resource intensity of a full k8s installation would be prohibitive. When paired with Terraform, the industry standard for building, changing, and versioning infrastructure, the provisioning process becomes entirely declarative. This synergy allows operators to define their desired state of virtual machines and network configurations in code, ensuring that staging and production environments are identical replicates, thereby eliminating the "it works on my machine" conflict that plagues manual deployments.

The Architectural Distinction Between k3s and k8s

Understanding the transition from standard Kubernetes (k8s) to k3s requires an analysis of the operational burdens associated with production-grade clusters. A standard k8s setup typically demands a rigorous separation of concerns to maintain stability and scalability.

The Control Plane Requirements

In a traditional k8s environment, the master nodes are dedicated to running the control plane. This involves managing the API server, the scheduler, and the controller manager. If these components share resources with the actual application workloads, a spike in application traffic could potentially crash the control plane, leading to a total cluster failure.

The etcd Database Layer

High Availability (HA) in k8s is traditionally achieved by deploying a separate cluster for the etcd database. etcd is a consistent and distributed key-value store used for all cluster data. Managing a standalone etcd cluster adds significant operational complexity, as it requires careful node sizing and networking to prevent data corruption or loss.

The Ingress Controller Strategy

To handle incoming external requests efficiently, best practices dictate the use of a separate node or a dedicated set of nodes to run the Ingress Controller. This ensures that the traffic routing layer does not compete for CPU and RAM with the business logic of the applications.

The k3s Solution

k3s resolves these pain points by focusing on a low-footprint design. Because it is distributed as a single small binary, it eliminates the need for extensive installation scripts and dependency management. This makes k3s not only suitable for remote virtual machines in the cloud but also portable enough to run on local development machines or resource-constrained hardware like the Raspberry Pi.

The Role of Terraform in Kubernetes Provisioning

Terraform serves as the foundational layer for this deployment, acting as the engine that translates human-readable configuration files into actual cloud resources. As an Infrastructure as Code tool, it allows for the automation of almost every operation related to Cloud Provider resources, providing a safety net through versioning and state management.

The Imperative vs. Declarative Gap

While Terraform is exceptional at creating the "shell" of the infrastructure—such as the Virtual Machine (VM) instances, disks, and network interfaces—it is not natively a configuration management tool. It is not intended to replace specialized tools like Puppet or Chef, which are designed for the internal state of a server. However, in a k3s deployment, Terraform bridges this gap using provisioners.

The Provisioning Lifecycle

To transform a raw Google Compute Engine VM into a functioning Kubernetes node, the server must be bootstrapped. There are several methods to achieve this:

• Immutable Infrastructure: Creating pre-configured machine images and rolling out servers from those images.
• Cloud Native Tools: Using tools like Ignition for early-stage configuration.
• SSH Automation: Connecting to the server via SSH and executing scripts.

The approach utilizing Terraform provisioners focuses on the third method, using a combination of remote and local execution to install the k3s binary and configure the local environment for management.

k3sup: Accelerating the Installation Process

k3sup is a specialized tool designed to simplify the installation of k3s clusters. It abstracts the complexities of SSH connectivity and script execution, allowing a user to provision a cluster with minimal manual intervention.

Installation of k3sup

To prepare the local environment for cluster orchestration, the k3sup executable must be installed. This is achieved through a direct shell command that downloads and installs the binary to a system directory:

bash curl -sLS https://get.k3sup.dev | sh sudo install k3sup /usr/local/bin/ k3sup version

The result of this installation is a command-line tool capable of several critical operations:

• Automated Installation: It can trigger the k3s installation on a remote server using SSH.
• KUBECONFIG Management: It automatically fetches the KUBECONFIG file from the newly created k3s cluster and saves it to the local machine. This file is essential because it provides the authentication and endpoint configuration needed for the kubectl command to communicate with the cluster.
• Node Expansion: It facilitates the rapid joining of worker nodes into an existing cluster via the k3sup join command.
• HA Configuration: While a single master is common for development, k3sup supports the creation of High Availability multi-master clusters.

Implementing the Infrastructure on Google Cloud Platform

Deploying k3s on Google Cloud requires a precise combination of network security and compute resources. The following components are mandatory for a successful deployment.

Firewall Configuration

Kubernetes clusters require specific ports to be open for communication between the control plane and the worker nodes. Specifically, port 6443 is required for the Kubernetes API server. To implement this in Terraform, a google_compute_firewall resource is defined and tagged to ensure it only applies to the relevant instances.

```hcl
resource "googlecomputefirewall" "k3s-firewall" {
name = "k3s-firewall"
network = "default"

allow {
protocol = "tcp"
ports = ["6443"]
}

target_tags = ["k3s"]
}
```

The impact of this configuration is that any VM instance tagged with "k3s" will allow incoming traffic on port 6443, which is the primary gateway for kubectl and other management tools to interact with the master node.

Master Node Provisioning

The master node is the heart of the cluster. In Google Compute Engine, this is represented by a google_compute_instance. The configuration requires a specific machine type, a boot disk with a compatible image (such as debian-9-stretch-v20200805), and a network interface that provides a public IP address.

The integration with k3sup occurs within the local-exec provisioner. Unlike other provisioners that run on the remote server, local-exec runs on the machine where Terraform is being executed. This allows Terraform to call the locally installed k3sup binary and pass the dynamic IP address of the VM it just created.

```hcl
resource "googlecomputeinstance" "k3smasterinstance" {
name = "k3s-master"
machine_type = "n1-standard-1"
tags = ["k3s", "k3s-master"]

bootdisk {
initializeparams {
image = "debian-9-stretch-v20200805"
}
}

networkinterface {
network = "default"
accessconfig {}
}

provisioner "local-exec" {
command = < k3sup install \
--ip ${self.networkinterface[0].accessconfig[0].natip} \
--context k3s \
--ssh-key ~/.ssh/googlecompute_engine \
--user $(whoami)
EOT
}

dependson = [
googlecompute_firewall.k3s-firewall,
]
}
```

In this block, the self construct is utilized to reference the nat_ip of the instance during its own creation process. The depends_on meta-argument ensures that the firewall is fully operational before Terraform attempts to run the k3sup installation, preventing connection timeouts.

Advanced Terraform Provisioning Techniques

Beyond the use of k3sup, Terraform offers built-in provisioners for those who prefer direct script execution over external binaries. These are primarily remote-exec and local-exec.

The remote-exec Provisioner

The remote-exec provisioner is used to execute a script on the remote server after it has been created. This is the traditional way of bootstrapping a server. The process involves specifying a connection block that tells Terraform how to access the server via SSH.

```hcl
provisioner "remote-exec" {
script = "${path.module}/scripts/bootstrap.sh"

connection {
user = "root"
host = "${self.accesspublicipv4}"
}
}
```

Key technical details of this implementation include:

• The path.module construct: This ensures that Terraform looks for the bootstrap.sh script relative to the directory where the module is located, making the code portable across different environments.
• The self reference: This allows the provisioner to dynamically grab the public IP address of the resource it is currently provisioning.
• The connection block: This establishes the SSH tunnel required to push and execute the script on the remote host.

The local-exec Provisioner

Complementing the remote execution, the local-exec provisioner is used to perform actions on the administrator's local machine. A common use case in Kubernetes deployments is the retrieval of the kubeconfig file. Without this file, the local kubectl client has no way of knowing where the cluster is located or how to authenticate with it.

hcl provisioner "local-exec" { command = "${path.module}/scripts/kubeconfig.sh ${self.access_public_ipv4}" }

This script typically copies the default kubeconfig from the remote master node to the local directory, enabling immediate verification of the cluster state.

Execution and Cluster Verification

Once the Terraform configuration is written, the deployment follows a standard three-step lifecycle.

The Deployment Sequence

1. terraform init: This command initializes the working directory, downloading the necessary providers (such as the Google Cloud provider) and setting up the backend.
2. terraform plan: This creates an execution plan, allowing the operator to review exactly which resources will be created, modified, or destroyed before any changes are applied to the cloud environment.
3. terraform apply: This executes the plan. Terraform provisions the VM instances, applies the firewall rules, and triggers the local-exec or remote-exec provisioners to install k3s.

Verifying the Cluster State

After the terraform apply command completes, the k3sup or local-exec process will have saved a kubeconfig file locally. By configuring kubectl to use this file, the operator can query the cluster to ensure all nodes are healthy and ready.

Running the following command provides a detailed view of the cluster topology:

bash kubectl get nodes

Expected output for a 4-node cluster would appear as follows:

This table confirms that the master node is functioning as the control plane and that the three worker nodes have successfully joined the cluster and are ready to accept workloads.

Comparative Analysis of Provisioning Methods

The choice between using k3sup and native Terraform provisioners depends on the specific needs of the project, ranging from rapid prototyping to strict enterprise configuration management.

Analysis of the k3s and Terraform Ecosystem

The integration of k3s and Terraform represents a significant shift in how Kubernetes is deployed for non-enterprise-scale use cases. The primary value proposition is the drastic reduction in the "time to first pod." By eliminating the manual overhead of etcd management and control plane separation, developers can focus on the application layer rather than the infrastructure layer.

The reliance on local-exec and remote-exec provisioners, while simple, highlights a critical architectural trade-off. These methods are "imperative" steps within a "declarative" tool. If a server is destroyed and recreated, Terraform must re-run these scripts to restore the k3s installation. This is why for larger-scale production environments, moving toward immutable infrastructure—where a VM image already contains k3s—is often the next logical step.

However, for the vast majority of tech enthusiasts, developers, and edge computing engineers, the combination of Terraform for infrastructure and k3sup for orchestration provides an unbeatable balance of speed and reproducibility. It allows for the creation of a production-ready cluster that can be mirrored across any VM provider, local server, or even a cluster of Raspberry Pis, ensuring that the environment remains consistent from the first line of code to the final deployment.

Sources

1. nimblehq
2. mkdev