Kubernetes Pod Eviction Mechanics

Kubernetes pod eviction is a critical system process where a pod assigned to a specific node is requested for termination. This mechanism is fundamentally a resource management strategy, ensuring that the overall health and stability of the cluster are maintained when system resources become constrained. In a production environment, the termination of a pod is not an arbitrary event but the result of specific triggers, ranging from the need to schedule higher-priority workloads to the exhaustion of hardware resources. When a pod is evicted, it is forced to abandon its current node; in some scenarios, these pods become nodeless, meaning they are no longer assigned to any physical or virtual machine within the cluster. This phenomenon is often exacerbated by the modern demands of high-performance computing, where the aggressive consumption of CPU and memory forces the Kubernetes scheduler and the kubelet to make rapid decisions about which workloads can remain operational.

Preemption and Priority-Based Eviction

Preemption occurs when the Kubernetes scheduler determines that a high-priority pod cannot be scheduled on any available node due to insufficient resources. To resolve this, the scheduler identifies pods with lower priority that are currently occupying the required resources and terminates them to make room for the higher-priority pod.

This process creates a hierarchy of importance within the cluster. The impact for the user is that critical applications remain available even during peak load, while non-critical or background tasks are sacrificed to ensure system stability. This connects directly to the concept of Pod Priority Classes, which allows administrators to explicitly define the importance of a workload.

The internal logic follows a strict priority path:

• High-priority pods are scheduled first.
• Lower-priority pods are preempted to free up CPU and memory.
• The scheduler ensures that the most vital services are never starved of resources by removing less critical ones.

Node-Pressure Eviction and Resource Thresholds

Node-pressure eviction is a proactive mechanism managed by the kubelet. The kubelet constantly monitors the resource levels of the node it resides on. When specific thresholds are breached—most commonly memory or disk space—the kubelet initiates the eviction process to prevent the node from crashing entirely due to out-of-memory (OOM) conditions or disk exhaustion.

The real-world consequence of node-pressure eviction is the prevention of a total node failure. If the kubelet did not evict pods, the underlying operating system might crash or become unresponsive, affecting every single pod on that node. This mechanism creates a safety buffer that maintains the operational integrity of the infrastructure.

Kubernetes categorizes pods into Quality of Service (QoS) classes to determine the order of eviction during node pressure:

• BestEffort: These pods have no requests or limits defined. They are the first to be evicted because they provide the least guarantee to the system.
• Burstable: These pods have requests defined, but their usage may exceed those requests. The kubelet ranks these in two sub-groups: those where usage exceeds requests are evicted before those where usage remains below requests.
• Guaranteed: These pods have requests and limits that are exactly equal. They are the most stable and are generally safe from eviction during scheduling. However, if system services require resources to prevent a crash, the kubelet will terminate Guaranteed pods, starting with those that have the lowest priority.

The following table outlines the ranking order for kubelet evictions:

For the user, this means that if containers are configured with very low resource requests, they are more likely to be categorized as BestEffort or low-tier Burstable, significantly increasing their probability of being evicted.

Taint-Based Eviction and Node Lifecycle

Taint-based eviction is a method of guiding pod placement and enforcing node restrictions. While taints generally prevent pods from being scheduled on a node, the NoExecute taint has a more aggressive effect: it evicts pods that are already running on the node if they do not possess a matching toleration.

A critical application of this is found in the node-lifecycle controller. The process operates as follows:

• Every kubelet reports a heartbeat every 10 seconds to the Kubernetes API server by updating a Lease resource.
• The node-lifecycle controller monitors this Lease.
• If a heartbeat is not received within 50 seconds (a configurable value), the controller sets the node condition to Unknown.
• The controller then adds an unreachable taint with the effect NoExecute.

By default, Kubernetes adds a toleration to every pod to ignore this NoExecute taint for 5 minutes. This prevents a momentary network flicker from causing a mass eviction event. However, once that 5-minute window expires, any pod not explicitly configured to tolerate the taint is immediately evicted.

The impact of this is severe because the pod deletion is forcible. This is equivalent to executing the command kubectl delete pod --force, which sets gracePeriodSeconds=0. Consequently, the pod does not respect its graceful termination period, leading to potential data loss or incomplete transactions.

Local Storage and Ephemeral Storage Eviction

Kubelet can trigger eviction based on the consumption of local storage. This typically happens when a pod exceeds its ephemeral storage usage, which includes logs or scratch filesystem writes, or when it exceeds the size limits configured on an emptyDir volume.

Unlike the forced eviction seen in some taint scenarios, local storage eviction allows the kubelet to terminate the pod gracefully. The pod is moved to a terminal phase based on the exit code of the process. This ensures that the pod has a chance to shut down correctly, although the end result is still the removal of the workload from the node to protect the node's disk space.

Administrative and API-Initiated Evictions

Beyond automatic system triggers, pods can be evicted through manual or API-driven requests.

• API-initiated eviction: Users can request an on-demand eviction of a pod on a specific node by utilizing the Kubernetes Eviction API. This is used for precise workload migration.
• Node drain: When a node becomes unusable or requires maintenance, administrators use the kubectl drain command. While kubectl cordon simply prevents new pods from being scheduled, kubectl drain nodename completely empties the node. This process evicts all pods while respecting their graceful termination periods.

Hard vs. Soft Evictions

Kubernetes differentiates between the severity of the eviction trigger through hard and soft thresholds.

• Hard Thresholds: These cause the kubelet to immediately terminate pods. There is no negotiation or delay once the threshold is hit.
• Soft Evictions: These also ignore the pod's graceful termination period and cap the grace period at a preconfigured value.

For administrators who require absolute control over pod lifecycles, it is possible to disable these hard and soft evictions within the kubelet configuration.

Monitoring and Alerting for Evictions

To maintain a stable cluster, monitoring pod evictions in real-time is mandatory. Prometheus is the standard tool for this purpose.

To identify all evicted pods in a cluster, the following Prometheus query is used:

kube_pod_status_reason{reason="Evicted"} > 0

To create a more sophisticated alerting system, this query can be paired with the following to identify pods that were evicted specifically after a failure:

kube_pod_status_phase{phase="Failed"}

By monitoring these metrics, operators can identify patterns—such as specific nodes consistently triggering evictions—and perform rightsizing of resource limits and requests to prevent future disruptions.

Conclusion

Pod eviction is an essential regulatory feature of Kubernetes, serving as the primary mechanism for managing finite hardware resources across a distributed system. Through the use of Preemption, Node-pressure evictions, and Taint-based logic, Kubernetes ensures that the most critical workloads are preserved while less essential pods are removed to prevent system-wide failure.

The analysis of these mechanisms reveals a complex dependency between resource configuration and pod stability. The Quality of Service (QoS) classes—BestEffort, Burstable, and Guaranteed—dictate the survival probability of a pod during a crisis. Furthermore, the interplay between the node-lifecycle controller and the NoExecute taint demonstrates how Kubernetes handles network partitions and node failures. While lares of pods are evicted daily due to the increasing demands of CPU and memory, these processes are not failures of the system, but rather the system functioning as designed to ensure overall cluster availability. Effective management of these evictions requires a combination of precise resource requests, strategic use of priority classes, and robust monitoring via tools like Prometheus to ensure that critical application disruptions are minimized.

Sources

1. Sysdig
2. Kubernetes Discuss
3. Ahmet.im