The Anatomy of the 503 No Healthy Upstream Error in Kubernetes and Istio Environments

The 503 Service Unavailable status code, specifically when accompanied by the message "no healthy upstream," represents one of the most critical and pervasive failure modes in modern cloud-native architectures. In the context of Kubernetes, service meshes like Istio, and reverse proxies such as Nginx or HAProxy, this error signals a breakdown in the communication chain between the ingress layer and the application logic layer. When a request enters a cluster, it passes through multiple layers of abstraction: the Load Balancer, the Ingress Controller (such as Istio Ingress Gateway or Nginx Ingress), and eventually the Service abstraction that routes traffic to individual Pods. A "no healthy upstream" error indicates that while the initial entry point (the proxy) is fully operational and capable of receiving traffic, it has found zero viable destinations to which it can forward the request. This failure is not a failure of the proxy itself, but rather a failure of the proxy's ability to find a functional backend. This state can emerge from a wide array of triggers, ranging from simple application crashes to complex networking issues within a service mesh's data plane, such as Istio's ambient mesh or ztunnel configurations.

Understanding the Mechanics of Upstream Failures

To diagnose the issue, one must first understand the role of an "upstream." In networking and proxy terminology, the upstream refers to the backend server or service that handles the actual application logic. The reverse proxy (the "downstream" from the perspective of the user) maintains a pool of these upstream endpoints. For a request to be successful, the proxy must select an endpoint from this pool that meets specific health and availability criteria.

When the proxy encounters a situation where the entire pool of endpoints is unavailable, it returns the HTTP 503 status code. The specific nuance of the "no healthy upstream" message implies that the proxy has a list of potential destinations, but every single one of them has failed a health check, has been removed from the service discovery mechanism, or is otherwise deemed unfit for traffic.

The impact of this error on an organization is significant. From the end-user's perspective, the service is completely unavailable, leading to a perceived total outage. From an SRE (Site Reliability Engineer) perspective, this error is often more complex than a simple "connection refused" because it implies the orchestration layer (Kubernetes) is technically communicating with the proxy, but the proxy has decided the backend is no longer capable of performing its duties.

Detailed Analysis of the Envoy Response Flag: UH

In environments utilizing Istio or Envoy-based proxies, the error is rarely just a 503 status code in the client's browser; it is recorded in the proxy's access logs with a specific "response flag." The response flag is a shorthand notation used by Envoy to provide rapid diagnostic telemetry regarding why a request failed.

The most critical flag to identify in these scenarios is UH.

• UH (No Healthy Upstream)
  The presence of the UH flag in the access log confirms that the proxy successfully matched the incoming request to a specific route and cluster, but upon attempting to select a backend from the load-balancing pool, it found no endpoints that were marked as healthy. This is the definitive signature of a 503 No Healthy Upstream error in an Istio or Envoy context.

It is vital to distinguish UH from other similar response flags to avoid wasted troubleshooting effort:

If a developer sees NR instead of UH, the troubleshooting path shifts entirely away from backend health and toward routing configuration, such as a misconfigured VirtualService or an incorrect host header.

Troubleshooting the Istio Ingress Gateway and Mesh

When an Istio Ingress Gateway returns a 503 with a UH flag, the investigation must move through the layers of the Istio control plane to the actual data plane components.

Step 1: Extracting Raw Gateway Logs

The first step in any Istio-related outage is to inspect the telemetry produced by the ingress gateway. This provides the ground truth of what the proxy "sees" when a request arrives.

To retrieve the last 30 lines of logs from the Istio Ingress Gateway, the following command is used:

bash kubectl logs -n istio-system deploy/istio-ingressgateway --tail=30

The operator must scrutinize the logs for the response_flags field. If UH is present, the gateway is functioning, but it has no valid destinations for the specific service being requested.

Step 2: Verifying Service and Endpoint Existence

Once UH is confirmed, the next logical step is to verify that the Kubernetes Service and its associated EndpointSlice or Endpoints actually exist and contain active IP addresses. A common cause for UH is that the Service is defined, but there are no Pods currently matching the Service's label selector.

To check if the service exists in the target namespace:

bash kubectl get svc <my-service> -n <production-namespace>

To inspect the actual backend IP addresses (Endpoints) that the Service is tracking:

bash kubectl get endpoints <my-service> -n <production-namespace>

For modern Kubernetes clusters, it is highly recommended to check the EndpointSlice resources, as they provide more granular data and are the preferred way to handle large-scale endpoint management:

bash kubectl get endpointslice -n <production-namespace> -l kubernetes.io/service-name=<my-service> -o jsonpath='{range .items[*].endpoints[*]}{.addresses}{" ready="}{.conditions.ready}{"\n"}{end}'

If the list of addresses is empty or the ready condition is false for all entries, the problem lies within the Pods themselves or their Readiness Probes.

Step 3: Examining Pod Health and Readiness Probes

If the Service exists but has no healthy endpoints, the investigation shifts to the Pod level. A Pod will be removed from a Service's endpoint list if its Readiness Probe fails.

To see the status of all pods in the namespace:

bash kubectl get pods -n <namespace> -o wide

To understand why a specific Pod is not considered ready, use the describe command:

bash kubectl describe pod <pod-name> -n <namespace>

The output of describe will contain an "Events" section. If you see repeated Readiness probe failed messages, the application is likely failing to respond on its designated health check port or path within the required time window. This can be caused by application deadlocks, high CPU/memory pressure, or incorrect configuration of the readinessProbe in the deployment manifest.

To test the health endpoint manually from within a running pod to rule out networking issues:

bash kubectl exec -it <pod-name> -n <namespace> -- curl -v http://localhost:8080/health

Advanced Failure Modes: Ambient Mesh and ztunnel Issues

A more complex and recent iteration of this error occurs in Istio Ambient Mesh environments. In the ambient model, traffic is handled by a shared component called ztunnel (Zero Trust Tunnel) rather than a sidecar proxy in every pod.

A specific issue documented in Istio's tracking systems involves communicating with the Kubernetes API from within a pod behind a ztunnel. In certain edge cases, particularly when the Kubernetes API server is accessed via its service IP (e.g., 10.43.0.1), the connection may be reset by the peer (Connection reset by peer).

In these scenarios, the error may manifest as:

bash wget: error getting response: Connection reset by peer

This can result in intermittent failures where 2 out of 4 pods might fail to reach the API, while others succeed. This pattern often points to a mismatch in how the ztunnel handles the identity or the encapsulation of the traffic when it attempts to bridge the gap between the pod's network namespace and the cluster's control plane.

General Causes of 503 Errors Across Other Platforms

The "no healthy upstream" phenomenon is not exclusive to Istio and Kubernetes; it is a fundamental concept in load balancing that applies to Nginx, HAProxy, and Docker Swarm.

Nginx Environment

In an Nginx configuration, this error appears in the error logs as:

text [error] no live upstreams while connecting to upstream

To diagnose this in Nginx, one must verify the upstream configuration and the status of the backend servers:

1. Check the Nginx error logs: tail -f /var/log/nginx/error.log
2. Verify the running configuration for upstream blocks: nginx -T | grep -A 20 "upstream"
3. Check the Nginx status module (if enabled) to see the state of upstreams: curl http://localhost/nginx_status

Docker Service Environment

In Docker Swarm, a service might fail with a similar logic if the task state is not healthy. A common symptom is:

text service "app" is not healthy

This indicates that the Docker daemon's health check mechanism has determined that the containers associated with the service are not meeting the required health criteria, causing the ingress routing layer to stop sending traffic to them.

Kubernetes Event Discrepancies

Sometimes, the "no healthy upstream" issue is caused by infrastructure-level constraints rather than application health. For example, if a deployment is scaled up, but the nodes themselves have constraints, the Pods will never reach a "Ready" state.

Example of a node-level failure:
text 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate

In this case, the load balancer is looking for endpoints, but the scheduler is unable to place the pods on nodes, leaving the service with zero available endpoints.

Systematic Troubleshooting Checklist

To resolve a 503 No Healthy Upstream error efficiently, follow this structured hierarchy of investigation:

1. Identify the Layer of Failure

   • Check the proxy/ingress controller logs.
   • Determine if the response flag is UH (No healthy upstream) or UF (Connection failure).

2. Verify the Service Abstraction

   • Ensure the Kubernetes Service exists.
   • Confirm that the Service has active Endpoints or EndpointSlices.
   • Use kubectl describe svc <name> to check for selector mismatches.

3. Assess Backend Pod Health

   • Check Pod status (kubectl get pods).
   • Review Pod events for Readiness Probe failures.
   • Verify that Pods are not stuck in CrashLoopBackOff or Pending states.
   • Manually test the application's health endpoint from within the container.

4. Investigate Network and Mesh Configuration

   • For Istio, verify that VirtualServices and DestinationRules are correctly configured.
   • Check for DNS resolution issues (e.g., CoreDNS failures) that might prevent the proxy from resolving the upstream hostname.
   • In Ambient Mesh, investigate ztunnel logs for connection resets when communicating with the Kubernetes API.

5. Check Infrastructure and Node Constraints

   • Verify that nodes are in a Ready state.
   • Ensure that Taints and Tolerations are correctly configured if pods are stuck in Pending.
   • Confirm that the underlying network provider (CNI) is functioning correctly and not dropping packets between nodes.

Conclusion: The Importance of Observability in Upstream Management

The "no healthy upstream" error is a symptom of a disconnected architecture. While it can be frustrating to encounter, it serves as a critical signal that the automated health-checking mechanisms of a modern orchestrator are performing their duty: they are preventing user traffic from being sent to a broken or non-existent destination.

A robust mitigation strategy involves moving beyond reactive troubleshooting. Developers should implement comprehensive observability by exporting Envoy's metrics to tools like Prometheus and visualizing them in Grafana. By monitoring the rate of UH response flags in real-time, teams can detect a service degradation—such as a failing deployment or a slow-leaking memory issue—before it escalates into a full-scale 503 outage. Furthermore, implementing sophisticated Readiness and Liveness probes that reflect the actual functional state of the application, rather than just the status of the process, is essential to ensure that the "upstream" pool remains truly healthy and capable of handling production-grade traffic.

Sources

1. Istio GitHub Issue #53306
2. OneUptime: How to Debug Istio Gateway 503
3. Alertmend: 503 No Healthy Upstream Error Guide
4. Uptrace: Understanding No Healthy Upstream