GitLab Kubernetes Agent Integration for CI/CD Pipeline Orchestration

The integration of kubectl within GitLab CI/CD pipelines represents a paradigm shift in how modern software engineering teams manage the lifecycle of containerized applications. By leveraging the GitLab Agent for Kubernetes, organizations can move away from the precarious management of static Kubeconfig files and long-lived credentials, transitioning instead toward a secure, scalable, and dynamic communication channel between the GitLab control plane and the Kubernetes cluster. This synergy allows for the execution of critical orchestration commands—such as kubectl apply and helm upgrade—directly from a pipeline, ensuring that the transition from code commit to production deployment is automated, auditable, and devoid of manual intervention.

The fundamental value proposition of this architecture lies in the ability to treat the cluster as a first-class citizen within the GitLab ecosystem. When a developer pushes code to a repository, the pipeline can now trigger specific actions within the cluster using the agent as a secure proxy. This is particularly vital when migrating from monolithic architectures—such as a Python Django DRF monolith—to a microservices architecture, where the number of deployments increases exponentially. Without a robust CI/CD pipeline integrated via the GitLab Agent, managing these disparate services would result in operational chaos. The goal is to establish a workflow where Continuous Integration (CI) validates the code and Continuous Deployment (CD) ensures that the validated changes are propagated to staging or production environments with absolute consistency.

Strategic Infrastructure Prerequisites

Before the execution of any kubectl commands within a GitLab pipeline, several foundational elements must be established to ensure connectivity and security.

The primary requirement is a GitLab repository hosting the codebase. This repository acts as the single source of truth for both the application code and the pipeline definition. Furthermore, a Kubernetes cluster must be connected to GitLab via the Agent for Kubernetes. This connection is not merely a network link but a bootstrapped installation of Flux, which enables a GitOps-driven approach to delivery.

For organizations utilizing GitLab Self-Managed instances, the configuration of Transport Layer Security (TLS) is a mandatory requirement. Failure to ensure the instance is configured with TLS can lead to critical connection failures during the pipeline execution. Specifically, when attempting to run kubectl commands, users may encounter errors indicating that the server requested credentials or that the certificate was signed by an unknown authority. These errors typically manifest as:

kubectl get pods
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Unable to connect to the server: certificate signed by unknown authority

In environments where the Kubernetes Agent Server (KAS) uses a self-signed certificate, the pipeline job may not trust the Certificate Authority (CA). The resolution for this failure requires configuring kubectl to trust the specific CA that signed the KAS certificate, thereby establishing a secure chain of trust.

The Mechanism of CI/CD Job Impersonation

A sophisticated feature of the GitLab Agent is its ability to impersonate the CI/CD job that accesses the cluster. This ensures that permissions are granular and tied to the specific execution context of the pipeline rather than a generic service account.

To implement this, the access_as key in the agent configuration must include the ci_job: {} key-value pair. When the agent forwards a request to the Kubernetes API, it generates impersonation credentials based on the job's identity.

The impersonation details are structured as follows:

• UserName: The username is formatted as gitlab:ci_job:<job id>. For example, a job with the ID 1074499489 becomes gitlab:ci_job:1074499489.
• Groups: The request is associated with the gitlab:ci_job group to identify it as a CI job. Additionally, it includes a comprehensive list of group and project IDs associated with the job.

For a job running in a path such as group1/group1-1/project1, where the root group has ID 23, the subgroup has ID 25, and the project has ID 150, the group list would encompass:
[gitlab:ci_job, gitlab:group:23, gitlab:group_env_tier:23:production, gitlab:group:25, gitlab:group_env_tier:25:production, gitlab:project:150, gitlab:project_env:150:prod, gitlab:project_env_tier:150:production]

This level of detail allows Kubernetes Role-Based Access Control (RBAC) to make highly specific decisions about what a pipeline job can or cannot do within the cluster.

Impersonation Metadata Mapping

The following table details the extra properties carried by the impersonated identity during a CI request:

Implementing Cluster Access with RBAC

To allow the CI/CD job to perform administrative or viewing tasks, specific ClusterRoleBindings must be created within the Kubernetes cluster. This prevents the "forbidden" errors that occur when a job attempts to modify cluster resources without sufficient permissions.

A common configuration involves creating a file named clusters/testing/gitlab-ci-job-flux-reconciler.yaml with the following content:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ci-job-admin
roleRef:
name: flux-edit-flux-system
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
subjects:
- name: gitlab:ci_job

kind: Group

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ci-job-view
roleRef:
name: flux-view-flux-system
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
subjects:
- name: gitlab:ci_job
kind: Group
```

This configuration ensures that any job identifying as gitlab:ci_job is granted the permissions defined in the flux-edit-flux-system and flux-view-flux-system roles, allowing the pipeline to manage Flux resources effectively.

Automating Container Registry Access

A critical step in the deployment pipeline is ensuring the Kubernetes cluster can pull images from the GitLab Container Registry. This requires the creation of a Docker registry secret within the cluster.

First, a deploy token must be created with the read_registry scope. The username and token associated with this deploy token should be stored as CI/CD variables in the GitLab project:
- CONTAINER_REGISTRY_ACCESS_TOKEN
- CONTAINER_REGISTRY_ACCESS_USERNAME

The following .gitlab-ci.yml configuration demonstrates how to automate the creation and deletion of this secret using the portainer/kubectl-shell:latest image.

```yaml
stages:
- setup
- deploy
- stop

create-registry-secret:
stage: setup
image: "portainer/kubectl-shell:latest"
variables:
AGENTKUBECONTEXT: my-group/optional-subgroup/my-repository:testing
beforescript:
- kubectl config use-context $AGENTKUBECONTEXT
script:
- kubectl delete secret gitlab-registry-auth -n flux-system --ignore-not-found
- kubectl create secret docker-registry gitlab-registry-auth -n flux-system --docker-password="${CONTAINERREGISTRYACCESSTOKEN}" --docker-username="${CONTAINERREGISTRYACCESSUSERNAME}" --docker-server="${CIREGISTRY}"
environment:
name: container-registry-secret
on_stop: delete-registry-secret

delete-registry-secret:
stage: stop
image: ""
variables:
AGENTKUBECONTEXT: my-group/optional-subgroup/my-repository:testing
beforescript:
- kubectl config use-context $AGENT_KUBECONTEXT
script:
- kubectl delete secret -n flux-system gitlab-registry-auth
```

In this workflow, the create-registry-secret job ensures that the cluster is authenticated to the registry before deployment begins. The on_stop trigger is utilized to clean up the secret when the environment is stopped, preventing the accumulation of stale credentials in the flux-system namespace.

Advanced Deployment Workflows with Flux and OCI

Combining the GitLab Agent with Flux allows for a sophisticated OCI-based deployment strategy. Instead of deploying raw manifests, the pipeline can build a container image, push it to the registry, and then instruct Flux to reconcile that image.

The following example illustrates a deployment for an NGINX application. Note the use of the fluxcd/flux-cli:v2.4.0 image and the specific environment mapping.

yaml nginx-deployment: stage: deploy variables: IMAGE_NAME: nginx-example IMAGE_TAG: latest MANIFEST_PATH: "./clusters/applications/nginx" IMAGE_TITLE: NGINX example AGENT_KUBECONTEXT: my-group/optional-subgroup/my-repository:testing FLUX_OCI_REPO_NAME: nginx-example NAMESPACE: flux-system environment: name: applications/nginx kubernetes: agent: $AGENT_KUBECONTEXT dashboard: namespace: default flux_resource_path: kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/nginx-example image: name: "fluxcd/flux-cli:v2.4.0" entrypoint: [""] before_script: - kubectl config use-context $AGENT_KUBECONTEXT script: - # Logic to build and push the OCI container to the GitLab container registry would go here

This configuration integrates the GitLab environment dashboard with the Kubernetes cluster. By specifying the flux_resource_path, users can track the status of the Kustomize resource directly within the GitLab UI, bridging the gap between the CI pipeline and the actual cluster state.

Troubleshooting Common Pipeline Failures

When implementing kubectl within GitLab CI, developers may encounter specific version-related or environment-related bugs.

One notable issue occurs with kubectl versions v1.27.0 and v1.27.1. Users may see the following error during manifest validation:

error: error validating "file.yml": error validating data: the server responded with the status code 426 but did not return more information

This error is the result of a bug in the shared Kubernetes libraries used by kubectl. The recommended resolutions are:
1. Use a different version of kubectl (upgrade or downgrade).
2. If the validation is not critical, disable it by adding the --validate=false flag to the command.

Additionally, the image used in the .gitlab-ci.yml file must be configured so that the directory used for Kubernetes configurations is writable. If the environment is read-only, kubectl may fail to write the temporary context needed to communicate with the agent.

Resource Lifecycle Management

Maintaining a clean cluster is as important as the deployment process itself. To remove resources and stop the deployment cycle, a systematic cleanup approach is required.

• Manifest Removal: To delete an application, such as the NGINX example, the corresponding YAML file (e.g., clusters/testing/nginx.yaml) should be deleted from the repository. Because Flux is monitoring the repository, it will automatically reconcile the state and remove the resources from the cluster.
• Secret Deletion: The container-registry-secret environment should be stopped. This action triggers the on_stop job, which executes the kubectl delete secret command, ensuring that no sensitive authentication data remains in the flux-system namespace.

Conclusion

The integration of kubectl within GitLab CI/CD via the Kubernetes Agent transforms the deployment process into a secure, automated pipeline. By utilizing job impersonation, organizations can enforce strict RBAC policies, ensuring that CI jobs operate with the minimum necessary privileges. The combination of Flux for GitOps and the GitLab Agent for direct command execution provides a hybrid approach that offers both the reliability of declarative state management and the flexibility of imperative command execution.

Whether migrating a Django DRF monolith to microservices or scaling a complex fleet of OCI-based applications, the key to success lies in the rigorous configuration of TLS, the correct mapping of AGENT_KUBECONTEXT, and the strategic use of ClusterRoleBindings. This architecture not only reduces the risk of human error but also provides a transparent, auditable path from code to production, fundamentally improving the velocity and stability of the software delivery lifecycle.

Sources

1. Get started deploying to Kubernetes - GitLab
2. CI/CD workflow - GitLab
3. Migrating a Python Django DRF Monolith to Microservices Part 4 - Ahmad W Khan