Kubernetes 1.22 Reaching New Peaks

The release of Kubernetes 1.22, themed "Reaching New Peaks," represents a pivotal moment in the evolution of the cloud-native ecosystem. This version is characterized by a historic volume of enhancements, a shift in release cadence, and a significant pruning of the API surface to ensure long-term stability and performance. The thematic essence of "Reaching New Peaks" is a testament to the resilience of the upstream community, which managed to deliver the highest number of enhancements in any release despite the global challenges of pandemic-induced burnout. For the end-user, this version marks a transition toward a more mature, secure, and performance-oriented orchestration platform.

One of the most critical aspects of this release is the aggressive removal of deprecated beta APIs. Kubernetes has moved toward a model where beta APIs are transitioned to stable (GA) versions, and once those stable versions are available, the beta versions are removed to prevent technical debt and confusion within the API server. This necessitates a proactive migration strategy for cluster administrators to avoid service disruptions. Furthermore, the integration of updated backend storage systems and the introduction of advanced security features—such as rootless control planes and default seccomp profiles—elevate the security posture of the entire cluster.

The support for Kubernetes 1.22 extends across various distribution models, including Amazon Elastic Kubernetes Service (Amazon EKS), Amazon EKS Distro, and Amazon EKS Anywhere. This broad support ensures that enterprise users can leverage the new features across managed and self-managed environments. From the implementation of Server-side Apply to the graduation of client credential plugins, Kubernetes 1.22 optimizes how developers interact with the API and how the system manages the internal state of the cluster.

API Removals and Migration Requirements

Kubernetes 1.22 implements a strict removal policy for several beta APIs. These APIs were previously deprecated in favor of more stable, GA (Generally Available) versions. The removal of these endpoints means that any manifest, Helm chart, or automation script relying on these specific API versions will fail upon deployment or upgrade to version 1.22.

The following table details the specific beta API removals and the corresponding stable migrations required:

The impact of these removals is most acutely felt in the Ingress resource. Users must migrate to the networking.k8s.io/v1 Ingress API, which has been available since version 1.19. This migration is not merely a version string change; it involves adapting to the stable API's structure. To complement this, the IngressClass resource is utilized, allowing administrators to configure and manage multiple types of Ingress controllers within a single cluster, providing a more flexible and scalable way to handle external traffic.

Security Enhancements and Rootless Architectures

Kubernetes 1.22 introduces several high-impact security features designed to reduce the attack surface of the cluster and limit the blast radius of potential compromises.

The introduction of default seccomp profiles for the kubelet is a primary security milestone. This is an alpha feature that provides cluster-wide seccomp defaults. When enabled via the appropriate command line flag and configuration, the system utilizes the RuntimeDefault seccomp profile instead of the Unconfined profile. By moving away from Unconfined, the system restricts the system calls a container can make to the Linux kernel, thereby enhancing the default security of the Kubernetes Deployment and providing peace of mind to security administrators.

Another critical alpha feature is the ability to run kubeadm control plane components as non-root users. Historically, control plane components required root privileges, which presented a significant security risk if a component were compromised. To activate this, administrators must enable the RootlessControlPlane feature gate. When deployed with this feature, the control plane operates with lower privileges, effectively implementing a long-requested security measure for the kubeadm ecosystem.

Furthermore, the management of service account tokens has evolved. The reduction of Secret-based Service Account Tokens is graduating to Beta. This shift is managed via the LegacyServiceAccountTokenNoAutoGeneration feature gate, which is set to true by default. Under this new paradigm, API credentials are obtained through the TokenRequest API. These tokens are stable as of version 1.22 and are mounted into Pods using a projected volume, eliminating the need for long-lived secrets that could be leaked or misused.

Storage and CSI Infrastructure

The storage landscape in Kubernetes 1.22 undergoes significant changes, particularly for Windows environments and the implementation of specific access modes.

Windows privileged containers are introduced as an alpha feature in this release. A key challenge for Windows nodes has been the use of CSI (Container Storage Interface) storage. To resolve this, CSIProxy has been introduced. CSIProxy allows CSI node plugins to be deployed as unprivileged pods. The proxy then handles the privileged storage operations on the node, ensuring that storage functionality does not require the plugins themselves to run with excessive privileges.

In the realm of volume access, the ReadWriteOncePod PersistentVolume access mode is graduating to Beta. This feature, controlled by the ReadWriteOncePod feature gate and enabled by default, restricts volume access to a single pod on a single node. This is a more granular control than the traditional ReadWriteOnce, which allows multiple pods on the same node to access the volume. This ensures strict isolation for specific workloads that cannot tolerate concurrent access.

Additional storage enhancements include:

• The introduction of nodeExpandSecret in the CSI PV source, which graduates to Beta. This allows the secretRef field to be passed to the CSI driver during a NodeExpandVolume operation, enabling dynamic secret management during volume expansion.
• The prevention of unauthorized volume mode conversion during volume restore. This feature adds a critical security layer to the VolumeSnapshot feature (which GA'd in 1.20), preventing unauthorized changes to the volume mode during restoration processes.

Core System Components and etcd

The reliability of a Kubernetes cluster depends heavily on its backing store, and Kubernetes 1.22 integrates updated versions of etcd to improve overall system health.

The default backend storage, etcd, is updated to version 3.5.0. This version brings several key improvements:

• Security and Performance: General enhancements to the efficiency and safety of the data store.
• Monitoring and Developer Experience: Improved tools for observing the health of the etcd cluster.
• Structured Logging: A migration to structured logging allows for better parsing and analysis of logs.
• Log Rotation: Built-in log rotation prevents disk space exhaustion on the control plane nodes.
• Traffic Overload Roadmap: The 3.5.0 release includes a detailed roadmap to address and mitigate issues related to traffic overload.

In the context of Amazon EKS, the team intended to ship with etcd 3.5.2, the recommended version for Kubernetes 1.22, due to the performance and reliability improvements found in the 3.5 series. However, the Amazon EKS team identified and verified two specific data inconsistency issues within etcd 3.5, highlighting the critical nature of the backing store's stability for enterprise-scale operations.

Resource Management and the cgroups v2 API

Kubernetes 1.22 introduces support for the cgroups v2 API as an alpha feature. Historically, Kubernetes relied on the v1 cgroups API, which limited the Quality of Service (QoS) class for a Pod primarily to CPU resources, such as cpu_shares.

The transition to cgroups v2 allows Kubernetes to exert much finer control over memory allocation and isolation. By utilizing the v2 API, the system can better manage how memory is distributed and limited across pods, preventing noisy neighbor effects and improving the overall stability of the node. This represents a fundamental shift in how the kubelet interacts with the Linux kernel to enforce resource constraints.

Declarative Configuration and Client Plugins

The user experience for managing resources is enhanced through the general availability of Server-side Apply.

Server-side Apply allows users and controllers to manage resources via declarative configurations. Instead of the client having to track the entire state of an object (the "read-modify-write" cycle), the client can simply send their fully specified intent. The server then handles the merging of this intent with the existing state. This feature has graduated to GA in 1.22, simplifying the development of controllers and the management of complex manifests.

Additionally, support for Kubernetes client credential plugins has graduated to stable. This feature, which has been in beta since version 1.11, provides:

• Improved support for plugins that require interactive login flows.
• A series of bug fixes to enhance stability.
• A sample implementation (sample-exec-plugin) to assist developers in creating their own custom credential plugins.

Cluster Deployment and Kubeadm

For those deploying clusters using kubeadm, Kubernetes 1.22 introduces several updates to the configuration and deployment process.

A new v1beta3 configuration API is now available for kubeadm. This version is the preferred API version, though v1beta2 remains available and is not yet deprecated. The v1beta3 API incorporates several long-requested features and deprecates older, obsolete configurations.

As mentioned previously, the RootlessControlPlane feature gate allows the kubeadm control plane components to run as non-root users, significantly reducing the risk profile of the control plane.

Upgrade Path and Version Transition

Upgrading to Kubernetes 1.22 and subsequently to later versions requires a strategic approach to avoid configuration failures.

A critical consideration for those moving from 1.22 to 1.23 is the removal of support for kube-scheduler configuration file version v1beta1. Administrators must update their kube-scheduler.yaml from apiVersion: kubescheduler.config.k8s.io/v1beta1 to apiVersion: kubescheduler.config.k8s.io/v1beta2 before proceeding with the upgrade to 1.23.

The recommended upgrade workflow involves:

• Patching the current minor version to the latest patch release (e.g., moving from 1.22.5 to 1.22.6).
• Reviewing the CHANGELOG for the target version to identify mandatory configuration changes.
• Applying configuration updates (such as the kube-scheduler API change) while still on the current minor version.
• Deploying the updated configuration to the control plane and worker nodes.
• Proceeding to the next major version upgrade.

Analysis of the Kubernetes 1.22 Impact

Kubernetes 1.22 is not merely a feature update; it is a structural realignment. The "Reaching New Peaks" theme manifests in the aggressive cleanup of the API. By removing beta APIs, the community is forcing a shift toward stable, predictable interfaces. This reduces the cognitive load for developers and minimizes the risk of breaking changes in subsequent releases.

The security trajectory of 1.22 is particularly impressive. The combination of default seccomp profiles, rootless control planes, and the movement away from Secret-based Service Account Tokens indicates a move toward a "Zero Trust" architecture within the cluster. The shift to TokenRequest API for credentials is a significant leap forward in securing pod-to-API communication.

From a performance perspective, the integration of etcd 3.5 and the alpha support for cgroups v2 indicate that Kubernetes is focusing on the "bottom-up" stability of the system. Better memory isolation and more efficient data storage directly translate to higher cluster density and better reliability for the end-user.

However, the transition is not without friction. The removal of Ingress beta APIs requires active intervention from cluster administrators. The necessity of migrating to networking.k8s.io/v1 means that any legacy automation that is not updated will break. This highlights the importance of the Kubernetes release cadence and the need for organizations to maintain a rigorous upgrade cycle.

In summary, Kubernetes 1.22 serves as a bridge between the experimental nature of earlier versions and the enterprise-grade stability required for the modern cloud. It balances the introduction of cutting-edge alpha features (like rootless control planes) with the graduation of essential tools (like Server-side Apply), ensuring that the platform remains flexible yet robust.

Sources

1. Amazon EKS now supports Kubernetes 1.22
2. Upcoming changes in Kubernetes 1.22
3. Kubernetes 1.22 release announcement
4. What's new in Kubernetes 1.27
5. Kubernetes upgrade notes 1.22-1.23