Orchestrating Kubernetes Deployments via GitLab CI/CD

The integration of GitLab CI/CD with Kubernetes creates a robust framework for delivering containerized applications with high velocity and minimal manual intervention. By leveraging GitLab's native pipeline capabilities alongside the orchestration power of Kubernetes, organizations can transition from manual deployments to a fully automated software delivery lifecycle. This synergy allows for the creation of traceable, consistent, and repeatable deployment patterns, ensuring that every change from a developer's commit to the production environment is audited and validated. The architecture typically follows a flow where GitLab CI triggers the build of a Docker image, pushes it to a Container Registry, and subsequently interacts with the Kubernetes API to update the cluster state. This process eliminates the "it works on my machine" syndrome by ensuring that the exact same image tested in staging is promoted to production.

Kubernetes Authentication and Access Management

Establishing a secure connection between the GitLab runner and the Kubernetes cluster is the most critical step in the deployment pipeline. Without proper authentication, the kubectl binary cannot communicate with the API server.

Service Account Implementation

The most secure and scalable method for authenticating GitLab CI is the use of a Kubernetes Service Account. A Service Account provides an identity for processes that run in a pod or external tools like GitLab runners.

To implement this, a specific manifest must be applied to the cluster. The service account gitlab-deploy is created in the default namespace to act as the primary identity for the CI pipeline.

yaml apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-deploy namespace: default

The impact of using a Service Account is the limitation of the blast radius. By creating a dedicated account for GitLab, administrators can ensure that the CI tool has only the permissions it needs, rather than using a high-privilege cluster-admin account.

Role-Based Access Control (RBAC) Configuration

A Service Account by itself has no permissions. It must be paired with a ClusterRole and a ClusterRoleBinding to define what the account can actually do within the cluster.

The ClusterRole named gitlab-deploy-role defines the specific API groups and resources the GitLab runner can manipulate.

yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: gitlab-deploy-role rules: - apiGroups: ["", "apps", "extensions"] resources: ["deployments", "services", "pods", "configmaps", "secrets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

The ClusterRoleBinding then connects the gitlab-deploy Service Account to the gitlab-deploy-role.

yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gitlab-deploy-binding subjects: - kind: ServiceAccount name: gitlab-deploy namespace: default roleRef: kind: ClusterRole name: gitlab-deploy-role apiGroup: rbac.authorization.k8s.io

This configuration provides a granular set of verbs, such as patch and update, which are essential for updating image tags during a deployment without deleting the entire deployment object.

Token Management and Variable Storage

Once the Service Account is established, a token must be generated to allow the GitLab runner to authenticate. This is achieved using the following command:

bash kubectl create token gitlab-deploy -n default --duration=8760h

The resulting token should be stored as a protected CI/CD variable in GitLab, typically named KUBE_TOKEN. Setting a long duration (such as 8760 hours) ensures that the pipeline does not fail due to token expiration, although shorter durations are preferred for higher security environments.

Pipeline Configuration for Cluster Connectivity

The .gitlab-ci.yml file must be configured to set up the Kubernetes context before any deployment commands are executed. This is often handled via a hidden job or a template that other jobs extend.

The Kube Setup Template

To avoid repeating authentication logic in every job, a .kube_setup template is utilized. This template uses the bitnami/kubectl:latest image, which comes pre-installed with the necessary binaries.

yaml .kube_setup: image: bitnami/kubectl:latest before_script: - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true - kubectl config set-credentials gitlab --token="$KUBE_TOKEN" - kubectl config set-context default --cluster=k8s --user=gitlab - kubectl config use-context default

The impact of this setup is the creation of a temporary kubeconfig within the runner's environment. This allows subsequent commands, like kubectl apply or kubectl set image, to target the correct cluster without requiring a physical file to be uploaded to the runner.

Alternative: Kubeconfig File Method

For organizations that prefer using a full kubeconfig file over a token, the file can be stored as a GitLab "File" type variable named KUBECONFIG_FILE. The pipeline then exports this variable to the environment:

yaml deploy: stage: deploy image: bitnami/kubectl:latest script: - export KUBECONFIG=$KUBECONFIG_FILE - kubectl get nodes - kubectl apply -f k8s/

This method is useful when the deployment requires complex context switching or multiple clusters.

Deployment Strategies and Manifest Management

There are several ways to translate a Docker image into a running set of pods in Kubernetes, ranging from simple manifest replacement to sophisticated package managers.

Simple Manifest Deployment with Sed

The most basic method involves using a placeholder in a YAML file and replacing it during the CI process using sed.

The Deployment Manifest (k8s/deployment.yaml):

yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp namespace: production spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: myapp image: IMAGE_PLACEHOLDER ports: - containerPort: 3000 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi

The CI Script:

yaml deploy: stage: deploy image: bitnami/kubectl:latest script: - sed -i "s|IMAGE_PLACEHOLDER|$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA|g" k8s/deployment.yaml - kubectl apply -f k8s/deployment.yaml - kubectl rollout status deployment/myapp -n production

The use of kubectl rollout status is critical as it prevents the pipeline from marking a job as "successful" before the pods are actually healthy and running.

Deployment via Kustomize

Kustomize allows for "overlay" management, enabling different configurations for staging and production without duplicating the entire manifest.

Base Kustomization (k8s/base/kustomization.yaml):

yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml

Production Overlay (k8s/overlays/production/kustomization.yaml):

yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base namespace: production images: - name: myapp newName: registry.gitlab.com/mygroup/myapp newTag: latest

The CI Implementation:

yaml deploy_production: stage: deploy image: bitnami/kubectl:latest script: - cd k8s/overlays/production - kustomize edit set image myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - kubectl apply -k . - kubectl rollout status deployment/myapp -n production

Deployment via Helm

Helm treats the application as a packaged "Chart," allowing for complex templating and versioning of the release.

Basic Helm Deployment:

yaml deploy: stage: deploy image: alpine/helm:3.13 script: - helm repo add myrepo https://charts.example.com - helm upgrade --install myapp myrepo/myapp \ --namespace production \ --set image.repository=$CI_REGISTRY_IMAGE \ --set image.tag=$CI_COMMIT_SHA \ --wait

The --wait flag in Helm ensures that the pipeline waits for the pods to reach a ready state, mirroring the behavior of kubectl rollout status.

Advanced Deployment Patterns

To minimize downtime and risk, advanced deployment patterns like Blue-Green and Canary releases are implemented.

Blue-Green Deployment

In a Blue-Green deployment, two identical environments exist. Only one (Blue) is live, while the other (Green) receives the new version for testing.

The Blue-Green CI Flow:

yaml deploy: script: - kubectl apply -f k8s/deployment-green.yaml - kubectl rollout status deployment/myapp-green -n production - ./scripts/smoke-test.sh green.internal.example.com - kubectl patch service myapp -n production -p '{"spec":{"selector":{"version":"green"}}}' - kubectl delete deployment myapp-blue -n production || true - kubectl label deployment myapp-green version=blue --overwrite

The impact of this strategy is near-zero downtime and the ability to perform an instantaneous rollback by simply switching the service selector back to the Blue deployment.

Canary Deployment

Canary deployments involve rolling out the new version to a small subset of users before a full rollout.

yaml deploy_canary: stage: deploy script: - kubectl apply -f k8s/deployment-canary.yaml - kubectl scale deployment myapp-canary --replicas=1 -n production - sleep 300 - ./scripts/check-error-rate.sh - kubectl scale deployment myapp-canary --replicas=3 -n production - kubectl scale deployment myapp-stable --replicas=0 -n production

This approach reduces the risk of a catastrophic failure by validating the new version's performance with a small percentage of real traffic.

The End-to-End Pipeline Architecture

A production-ready pipeline integrates building, testing, and deploying across multiple environments.

Pipeline Variables and Global Setup

The pipeline uses global variables to maintain consistency across jobs.

```yaml
variables:
IMAGETAG: $CIREGISTRYIMAGE:$CICOMMITSHA
DOCKERHOST: tcp://docker:2376
DOCKERTLSCERTDIR: "/certs"

stages:
- build
- test
- deploy
- cleanup
```

Build and Test Phases

The build stage utilizes Docker-in-Docker (dind) to build and push images to the GitLab registry.

```yaml
build:
stage: build
image: docker:24.0
services:
- docker:24.0-dind
script:
- docker login -u $CIREGISTRYUSER -p $CIREGISTRYPASSWORD $CIREGISTRY
- docker build -t $IMAGETAG -t $CIREGISTRYIMAGE:latest .
- docker push $IMAGETAG
- docker push $CIREGISTRY_IMAGE:latest

test:
stage: test
image: $IMAGE_TAG
script:
- npm test
```

Multi-Environment Deployment

Deployments are split between staging and production, with different triggers for each.

Staging Deployment:

yaml deploy_staging: extends: .kube_setup stage: deploy script: - kubectl create namespace staging || true - | cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: myapp namespace: staging spec: replicas: 2 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: myapp image: $IMAGE_TAG ports: - containerPort: 3000 EOF - kubectl rollout status deployment/myapp -n staging --timeout=300s environment: name: staging url: https://staging.example.com only: - develop

Production Deployment:

yaml deploy_production: extends: .kube_setup stage: deploy script: - kubectl set image deployment/myapp myapp=$IMAGE_TAG -n production - kubectl rollout status deployment/myapp -n production --timeout=300s environment: name: production url: https://example.com when: manual only: - main

The when: manual keyword for production ensures a human-in-the-loop gate, preventing accidental deployments to the live environment.

Reliability and Pipeline Optimization

Ensuring the stability of the deployment process requires implementing rollback mechanisms and optimizing resource usage.

Rollback Mechanisms

When a deployment fails, the ability to revert to the previous stable state is paramount. This is handled via the kubectl rollout undo command.

```yaml
rollbackproduction:
extends: .kubesetup
stage: cleanup
script:
- kubectl rollout undo deployment/myapp

rollbackuserservice:
stage: deploy
script:
- kubectl rollout undo deployment/user-service
when: on_failure
```

The when: on_failure trigger allows GitLab to automatically initiate a rollback if the primary deployment job fails, significantly reducing the Mean Time to Recovery (MTTR).

Performance Optimization

To speed up the pipeline, GitLab provides caching and artifact management.

Caching Dependencies:

yaml cache: paths: - .venv/ - node_modules/

Artifact Management:

yaml artifacts: paths: - build/ expire_in: 1 week

Caching ensures that dependencies are not re-downloaded on every run, while artifacts store build outputs that may be needed by subsequent stages (e.g., the test stage using a binary produced in the build stage).

Summary of Technical Specifications

The following table outlines the technical requirements and configurations for the GitLab to Kubernetes integration.

Operational Best Practices

For a secure and maintainable pipeline, several organizational standards should be applied.

Branch Protection and Environment Control

Deployments should be restricted based on the branch. The main branch or specific tags should be the only sources for production deployments. Feature branches can utilize dynamic environments for review.

yaml environment: name: review/$CI_COMMIT_REF_NAME url: https://$CI_ENVIRONMENT_SLUG.example.com

Secure Variable Management

Environment-specific variables (database URLs, API keys) must never be hardcoded in the .gitlab-ci.yml file. Instead, they should be stored in GitLab Settings > CI/CD > Variables and marked as "Masked" and "Protected" to prevent them from appearing in job logs.

Monitoring and Observability

The pipeline status is monitored via the GitLab CI/CD visual dashboard. This allows operators to:
- Navigate to CI/CD > Pipelines to see the overall flow.
- Inspect individual job logs to identify the exact point of failure.
- Use kubectl rollout status to verify the health of the pods within the cluster.

Conclusion

The implementation of a GitLab CI/CD pipeline for Kubernetes deployments transforms the delivery process from a series of manual steps into a choreographed stream of automated events. By utilizing a combination of Service Accounts for secure authentication, kubectl or Helm for manifest application, and advanced strategies like Blue-Green or Canary releases, organizations can achieve a high level of deployment confidence. The integration of automated rollbacks and strict branch protection ensures that only validated code reaches production, while the use of caching and artifacts optimizes the pipeline for speed. Ultimately, this architecture provides a scalable foundation for microservices, enabling teams to deploy frequently and reliably while maintaining full traceability of every change.

Sources

1. OneUptime - Deploy Kubernetes GitLab CI
2. Ahmad W Khan - Migrating Python Django DRF Monolith to Microservices