Kubernetes Docker Internal DNS Resolution

The architectural intersection of local containerization and orchestration often manifests as a critical networking challenge when a CI/CD pipeline attempts to communicate with a local Kubernetes cluster. Specifically, the hostname kubernetes.docker.internal serves as a specialized DNS entry designed to facilitate communication between the host machine and the Kubernetes API server provided by Docker Desktop. In a standard local environment, this hostname is resolved automatically by the Docker Desktop DNS resolver, allowing developers to interact with their cluster via kubectl from the host terminal without manual IP configuration. However, this seamless resolution fails catastrophically when the execution context shifts from the host machine to a containerized CI/CD runner, such as a GitLab Runner executing a job in an Alpine-based image.

The failure of kubernetes.docker.internal resolution is not a failure of the Kubernetes cluster itself, but rather a limitation of the DNS scope within the containerized runner. When a job runs inside a Docker container, it possesses its own isolated network namespace and its own /etc/resolv.conf configuration. The internal DNS resolver of the runner container does not inherently know the mapping for kubernetes.docker.internal because that mapping is a convenience provided by the Docker Desktop host process to the host OS. Consequently, when a curl command or a kubectl call is initiated from within the runner, the request is sent to the configured DNS server, which returns a "Could not resolve host" error. This disconnection prevents the pipeline from executing essential before_script validations and blocks the deployment of Kubernetes manifests, such as namespaces and services, effectively stalling the delivery pipeline.

The Anatomy of the DNS Resolution Failure

The technical failure manifests during the deploy stage of a CI/CD pipeline, specifically when utilizing a container image such as lachlanevenson/k8s-kubectl:latest. The sequence of failure occurs during the execution of a curl command intended to verify connectivity to the Kubernetes API server.

• The Direct Fact: A GitLab CI pipeline fails during the before_script phase when executing curl -k https://kubernetes.docker.internal:6443, resulting in the error curl: (6) Could not resolve host: kubernetes.docker.internal.
• Impact Layer: For the developer, this means the pipeline cannot verify the availability of the cluster before attempting to apply configurations. This results in a job failure that prevents the deployment of the application's namespace.yml and service.yml files, meaning the application is never updated in the target environment.
• Contextual Layer: This failure is paradoxical because the developer may find that kubectl works perfectly from their local terminal. This is because the host terminal is outside the container's network isolation and leverages the Docker Desktop DNS resolution, whereas the GitLab Runner is trapped within a container that lacks this specific host-mapping.

The environmental context of this failure is defined by a specific set of versions and configurations:

Docker Desktop Kubernetes Integration

To understand why kubernetes.docker.internal exists, one must analyze the integration of Kubernetes within Docker Desktop on Windows and macOS. Docker Desktop provides a streamlined, single-node Kubernetes cluster that is integrated directly into the Docker engine.

• The Direct Fact: Enabling Kubernetes is achieved via Docker Desktop -> Options -> Kubernetes -> Enable Kubernetes.
• Impact Layer: This action installs a fully functional 1-node local Kubernetes cluster. It removes the need for the user to manually set up a complex cluster using tools like kubeadm or Minikube, significantly reducing the barrier to entry for local development.
• Contextual Layer: Once enabled, Docker Desktop creates a virtualized network bridge. The hostname kubernetes.docker.internal is registered within this bridge to point to the API server of the internal Kubernetes node. This is why the developer's local terminal can reach the server, as it resides on the same host network.

Further configuration of this local environment involves several initialization scripts and services:

• init-kubernetes-dashboard.sh: This script installs the Kubernetes Dashboard and configures the necessary Role-Based Access Control (RBAC) authorization.
• init-ingress-nginx.sh: This script installs the Nginx Ingress controller and sets it as the default ingress controller for the cluster.
• kubectl proxy or run-proxy.sh: Running the proxy as a daemon allows the user to access the Kubernetes Dashboard via a local browser at http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/node?namespace=_all.
• get-bearer-token.sh: This utility generates an actual bearer token, stored in ./tokens/bearer-token.txt.user, which is required for authenticating with the dashboard.

GitLab CI Pipeline Configuration and Failure Points

The failure occurs within a structured GitLab CI YAML configuration. The pipeline is divided into two primary stages: build and deploy.

The build_image stage is successful. It utilizes docker:27.0.0-rc.1-alpine3.20 and the docker:27.0.0-rc.1-dind-alpine3.20 service. The script performs the following operations:

• docker login -u $REGISTRY_USER -p $REGISTRY_PASSWORD
• docker-compose build
• docker tag portfolioapp-frontend $IMAGE_NAME_FRONTEND:$IMAGE_TAG
• docker tag portfolioapp-backend $IMAGE_NAME_BACKEND:$IMAGE_TAG
• docker push $IMAGE_NAME_FRONTEND:$IMAGE_TAG
• docker push $IMAGE_NAME_BACKEND:$IMAGE_TAG

This stage succeeds because it only interacts with the Docker Registry and the local Docker-in-Docker (DinD) environment, which does not require resolution of the Kubernetes API server.

The deploy stage is where the failure occurs. The configuration for this stage is as follows:

• Variables used include KUBE_TOKEN, CI_KUBE_TOKEN, KUBE_CONTEXT (set to my-production-context), AGENT_ID, AGENT_TOKEN, KAS_URL, and K8SPROXY_URL (set to https://kubernetes.docker.internal:6443).
• The before_script contains the problematic sequence:
  1. apk --no-cache add curl
  2. curl -k https://kubernetes.docker.internal:6443
• The failure of the curl command stops the execution of the subsequent kubectl configuration commands, including:
  • kubectl config set-credentials gitlab-ci-serviceaccount --token=$CI_KUBE_TOKEN
  • kubectl config set-context my-production-context --cluster=my-production-cluster --user=gitlab-ci-serviceaccount --namespace=prod
  • kubectl config use-context my-production-context
  • kubectl config set-credentials agent:$AGENT_ID --token="ci:${AGENT_ID}:${AGENT_TOKEN}"
  • kubectl config set-cluster gitlab --server="${KAS_URL}"
  • kubectl config set-context "$KUBE_CONTEXT" --cluster=gitlab --user="agent:${AGENT_ID}"
  • kubectl config use-context "$KUBE_CONTEXT"

The script section, which is never reached due to the before_script failure, is intended to execute:

• kubectl config current-context
• kubectl config get-contexts
• kubectl apply -f portfolio_kubernetes/namespace.yml
• kubectl apply -f portfolio_kubernetes/service.yml

GitLab Agent for Kubernetes Installation

To bridge the gap between GitLab and the Kubernetes cluster, the GitLab Agent is employed. The installation is performed via Helm, which is the standard package manager for Kubernetes.

The installation command used is:
helm upgrade --install portfolioagent gitlab/gitlab-agent --namespace gitlab-agent-portfolioagent --create-namespace --set image.tag=v17.2.0-rc1 --set config.token=glagent-mytoken --set config.kasAddress=wss://kas.gitlab.com --set config.gitlabUrl=https://gitlab.com./ --set-file config.kasCaCert=tls.crt

This deployment establishes a connection between the cluster and the GitLab KAS (Kubernetes Agent Server) at wss://kas.gitlab.com. While the agent is successfully connected and the developer can reach the Kubernetes server from their local terminal, the CI pipeline remains unable to resolve the kubernetes.docker.internal host.

Technical Analysis of Network Pathing

The discrepancy between the terminal's success and the pipeline's failure is rooted in the networking layers of the environment.

1. Host Terminal Path:
   Host Terminal -> Docker Desktop DNS Resolver -> Mapping for kubernetes.docker.internal -> Kubernetes API Server (IP: 127.0.0.1 or internal VM IP).

2. GitLab Runner Path:
   GitLab Runner Container -> Container DNS Resolver (/etc/resolv.conf) -> DNS Server (e.g., Google DNS or Docker DNS) -> No mapping for kubernetes.docker.internal -> NXDOMAIN (Non-Existent Domain).

The runner is attempting to resolve a name that is only valid within the host's context. To resolve this, the runner would need an explicit mapping of the hostname to the IP address of the Kubernetes API server, or the pipeline must be configured to use a reachable IP address instead of the kubernetes.docker.internal alias.

Conclusion: Analysis of the Resolution Failure

The error curl: (6) Could not resolve host: kubernetes.docker.internal is a classic manifestation of a DNS scope mismatch. The developer is attempting to use a host-level convenience hostname inside a containerized environment that does not share the host's DNS resolver. While Docker Desktop facilitates local development by providing this alias, it does not inject this alias into every container spawned by a CI/CD runner.

The failure is compounded by the use of before_script as a validation gate. Because the curl command is placed before the kubectl configuration, the pipeline terminates before it can utilize the KAS_URL or other connectivity methods that might be functioning. The fact that the GitLab Agent is connected and kubectl works on the host proves that the cluster is healthy and accessible; the issue is strictly limited to the network namespace of the GitLab runner.

To rectify this, the pipeline must stop relying on kubernetes.docker.internal for its connectivity checks. Instead, it should utilize the actual IP address of the API server or leverage the GitLab Agent's connectivity (KAS) which is already configured and functional. The reliance on an internal Docker Desktop hostname for a pipeline intended for Digital Ocean deployment indicates a configuration drift where local development shortcuts are being applied to a remote deployment strategy. For a production-ready pipeline, the K8SPROXY_URL and curl checks should be pointed toward the public endpoint of the Digital Ocean Kubernetes cluster rather than a local Docker Desktop internal address.