K3s-Driven GitLab Infrastructure Orchestration

The intersection of lightweight Kubernetes distributions and comprehensive DevOps platforms creates a powerful synergy for modern software development. By leveraging k3s to host a GitLab instance, developers and organizations can implement a production-grade orchestration layer without the overhead of a massive compute cluster. k3s is a slimmed-down, fully functional Kubernetes distribution, packaged as a 40MB binary. This architectural efficiency allows it to run with as little as 512MB of RAM, making it an ideal candidate for local development environments, small-scale organizational tools, or as a bridge to full-scale production clusters.

Containers have fundamentally altered the portability of applications. GitLab, being containerized, can be moved across environments with ease. However, Kubernetes elevates this portability by providing the management plane necessary to handle scaling, self-healing, and deployment. k3s makes this portability accessible at the smallest possible scale, allowing for a single-node setup that remains compliant with Kubernetes standards. This approach is particularly beneficial for those who want to achieve uniformity in systems deployment, monitoring, and management across all development operations without managing a complex, multi-node cluster.

Implementing GitLab via k3s allows a team to unify their management plane under the Kubernetes API. This provides a strategic advantage: the specific infrastructure vendor—whether it be Digital Ocean, Linode, Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure—becomes a high-level business concern rather than a technical limitation. Because Kubernetes is cloud-native and vendor-neutral, a well-implemented k3s cluster ensures that the migration between these providers is seamless. For local development, k3s stands as a more viable alternative to minikube because it is closer to a production-style deployment, offering a more realistic simulation of how an application will behave in a live environment.

K3s Core Architectural Specifications

The deployment of k3s as a foundation for GitLab is rooted in its ability to provide a fully compliant Kubernetes experience within a minimal footprint. This makes it accessible for various hardware profiles, from low-power local machines to high-performance cloud virtual machines.

The impact of these specifications is a drastic reduction in the barrier to entry for self-hosting GitLab. Instead of requiring a cluster of high-specification servers, an organization can initiate their DevOps lifecycle on a single custom node. This ensures continuity in development operations, as the same API calls and configuration patterns used in the k3s environment will translate directly to larger Kubernetes environments.

Local DNS Server Configuration for GitLab

A critical component of a self-hosted GitLab instance is the ability to resolve domain names. To ensure that the GitLab instance is accessible via a friendly URL (e.g., gitlab.dev.davar.com) rather than just an IP address, a local DNS server must be configured. The standard tool for this is BIND9.

To begin the installation of the necessary DNS utilities, the following command is executed:

sudo apt-get install bind9 bind9utils bind9-doc dnsutils

The configuration of BIND9 involves several files to define how the server handles queries and where the zone files are located. The named.conf.options file defines the general behavior of the DNS server.

options { directory "/var/cache/bind"; auth-nxdomain no; # conform to RFC1035 listen-on-v6 { any; }; listen-on port 53 { any; }; allow-query { any; }; forwarders { 8.8.8.8; }; recursion yes; };

The named.conf.local file is where specific zones are defined. In this configuration, the zone davar.com is established as a master zone, and the reverse lookup zone is defined to map IP addresses back to hostnames.

zone "davar.com" { type master; file "/etc/bind/forward.davar.com"; }; zone "0.168.192.in-addr.arpa" { type master; file "/etc/bind/reverse.davar.com"; };

The reverse data file (reverse.davar.com) provides the necessary BIND reverse data for the local loopback interface, ensuring that internal network communication is routed correctly.

$TTL 604800 @ IN SOA davar.com. root.davar.com

Implementing this DNS layer ensures that the GitLab instance is integrated into the local network properly, preventing connectivity issues that often arise when relying solely on /etc/hosts files, which are not scalable across multiple nodes.

K3s Installation and Cluster Management

Installing k3s is designed to be a streamlined process. For environments running on cloud VMs or bare-metal lab servers, the installation can be initiated via a curl script.

To define the version and a secure cluster secret, the following variables are exported:

export INSTALL_K3S_VERSION=v1.16.10+k3s1
export K3S_CLUSTER_SECRET=$(head -c48 /dev/urandom | base64)

The actual installation is performed with the following command:

curl -sfL https://get.k3s.io | sh -s – server

Once the server is running, the kubectl configuration file, located at /etc/rancher/k3s/k3s.yaml, must be managed. For users accessing the cluster from a local workstation, this file is downloaded via SCP:

scp root@VM_PUBLIC_IP_OR_BARE_METAL_IP:/etc/rancher/k3s/k3s.yaml ~/.kube/k8s-c1

To ensure the configuration points to the correct Kubernetes API and utilizes proper naming for clusters and contexts, the file is modified using sed:

sed -i .bk "s/default/k8s-c1/" ~/.kube/k8s-c1
sed -i .bk "s/127.0.0.1/FQDN/" ~/.kube/k8s-c1

The FQDN is replaced with the actual Fully Qualified Domain Name, such as sofia1-1.c1.example.com. To use this specific configuration in a terminal session, the KUBECONFIG environment variable is set:

export KUBECONFIG=~/.kube/k8s-c1

For newer versions of k3s (where Kubernetes is greater than 1.20), the installation process remains similar. A user might SSH into the server and run:

ssh [email protected] -o IdentitiesOnly=yes

Followed by the installation command:

curl -sfL https://get.k3s.io | sh -

The K3S_TOKEN is a vital security component required for adding worker nodes to the cluster. This token is stored on the master server at the following path:

/var/lib/rancher/k3s/server/node-token

Advanced Cluster Setup with k3sup and Kilo

For users seeking a more automated approach to cluster management, k3sup offers a way to set up both the cluster and the local kubectl configuration simultaneously. This eliminates the need to manually handle tokens.

To install k3s on a remote server using k3sup, the following command is used:

k3sup install --ip $IP --user root

This process assumes that SSH keys have already been added to the server. Once installed, the user can verify the node status with:

kubectl get nodes

If the infrastructure requires more than a single node, worker agents can be joined to the cluster without manually retrieving the token, as k3sup handles this automatically:

k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER

In scenarios where a hybrid cluster is required across different networks, Kilo can be utilized. Kilo provides a VPN setup that allows nodes to report as "Ready" even when separated by different network boundaries. The process requires the kubectl config located at /etc/rancher/k3s/k3s.yaml on each node.

On the master node, the file is generated automatically. For worker nodes, the file must be modified to point to the master's FQDN and distributed:

ssh [email protected] sed "s/127.0.0.1/master.c2.example.com/" /etc/rancher/k3s/k3s.yaml >k3s.yaml

After distributing the modified k3s.yaml to /etc/rancher/k3s/k3s.yaml on all nodes, Kilo is installed by applying the configuration manifest:

kubectl apply -f https://raw.githubusercontent.com/squat/kilo/master/manifests/kilo-k3s.yaml

GitLab Integration and Configuration

Once the k3s cluster is operational, the GitLab instance can be deployed, often using Helm charts. The integration of the k3s cluster into the GitLab management plane requires providing the API URL, CA Certificate, and Service Token. For instance, an API URL might look like https://192.168.99.102:8443.

A common issue encountered during this integration is the error: "https://dev-k3s.davar.com:6443 is blocked: Requests to the local network are not allowed". This occurs because GitLab's default security settings block outbound requests to the local network to prevent certain types of attacks.

To resolve this, an administrator must:

• Log in to GitLab with the admin account.
• Navigate to "settings" -> "network" -> "Outbound requests".
• Check the box labeled "Allow requests to the local network from web hooks and services".
• Click "Save changes".

Additionally, it is imperative to ensure that Role-Based Access Control (RBAC) is enabled within the Kubernetes configuration to maintain security and granular permission management.

CI/CD Pipeline Connectivity and GitOps

The primary value of hosting GitLab on k3s is the ability to integrate the CI/CD pipeline directly with the cluster. While GitLab provides a native Kubernetes integration, a more generic approach can be adopted to ensure the pipeline is portable across different CI/CD platforms.

The core mechanism for this is the use of kubectl within the CI/CD pipeline's deployment step. By utilizing kubectl, the pipeline can manage the cluster's state directly. This enables the transition toward GitOps, a process popularized by Weaveworks.

GitOps shifts the focus from manual deployments to a state where the Kubernetes cluster matches the state described by configuration files residing in a Git repository. In a GitOps workflow:

• The system reacts to git push events.
• Configuration files in the Git repository act as the single source of truth.
• The objective is to replace the manual kubectl apply command with a simple git push.

This automation ensures that the environment is reproducible and that the active configuration state is always synchronized with the version-controlled manifests.

Component Comparison for Development Environments

When choosing between k3s and minikube for GitLab development, the decision rests on the desired proximity to production.

For those who require a simple cluster consisting only of a k3s server without additional agents, the installation of CoreDNS and MetricServer is typically recommended to ensure proper service discovery and resource monitoring.

Detailed Analysis of Infrastructure Continuity

The implementation of GitLab on k3s is not merely a technical exercise in reducing resource consumption; it is a strategic move toward infrastructure continuity. In traditional development cycles, there is often a "parity gap" between the local development environment and the production cluster. This gap leads to the "it works on my machine" syndrome, where code behaves differently once deployed to a full-scale Kubernetes environment.

By using k3s, the developer employs a "fully compliant production-grade Kubernetes distribution." This means the manifests, service definitions, and ingress controllers used in the k3s-hosted GitLab instance are identical to those that would be used in a massive cluster. The impact is a seamless transition; the only change is the scale of the compute resources.

Furthermore, the ability to deploy on a single custom node for local development, while maintaining the option to expand to cloud providers like AWS or GCP, allows organizations to start lean. They can validate their CI/CD flows and GitOps strategies on a low-cost VM before committing to expensive cloud infrastructure. The use of k3sup and Kilo further extends this flexibility, allowing for the creation of hybrid clusters that bridge the gap between local lab servers and remote cloud nodes.

The integration of GitLab as the central hub for development operations, combined with k3s as the orchestration layer, creates a feedback loop. The GitLab Runner, installed via Helm, can trigger deployments that are immediately applied to the k3s cluster. This allows for rapid iteration and testing of the deployment pipeline. When combined with GitOps, the entire lifecycle—from code commit to production deployment—is managed through a single Git interface, reducing the risk of configuration drift and increasing the reliability of the release process.

Sources

1. k3s-GitLab-development
2. Install GitLab Community Edition to k3s
3. Setting up a simple CI/CD flow with k3s and GitLab