Kubernetes 1.31 Release Notes (OBT)

Indexes

• New and Enhanced Features
• API Changes and Removals
• References

New and Enhanced Features

Kubernetes 1.31

• Start ordinal of a StatefulSet

  StatefulSet start ordinal moved to the General Availability (GA) state in Kubernetes 1.31. By default, each pod in a StatefulSet is assigned an integer ordinal from 0. With this feature, you can configure a start ordinal for each pod. For details, see Start ordinal.

• Elastic indexed jobs

  Elastic indexed jobs moved to GA in Kubernetes 1.31. You can scale indexed Jobs up or down by modifying fields .spec.completions and .spec.parallelism. For details, see Elastic Indexed Jobs.

• Pod failure policy

  Pod failure policies moved to GA in Kubernetes 1.31. This feature helps you handle pod failures based on the container exit codes and pod conditions. For details, see Pod failure policy.

• Pod disruption conditions

  Pod disruption conditions moved to GA in Kubernetes 1.31. The new DisruptionTarget condition indicates that the pod is about to be deleted due to a disruption. The reason field indicates one of the following reasons for the pod termination: preempted by a pod with a higher priority, the pod has been cleared due to node deletion, or the pod is terminated by kubelet. When a pod is created using a job or CronJob, you can use these pod disruption conditions as part of your job's pod failure policy to define the action when a pod is abnormal. For details, see Pod disruption conditions.

• Selectable fields for custom resources

  Selectable fields for custom resources moved to Beta in Kubernetes 1.31. You can specify the selectableFields field of a CustomResourceDefinition to define which other fields in a custom resource may be used in field selectors. Field selectors can then be used to get only resources by filtering List, Watch, and DeleteCollection requests. For details, see Selectable fields for custom resources.

• Job success policy

  Job success policies moved to Beta in Kubernetes 1.31. When creating an indexed Job, you can define when a job can be declared as succeeded, based on the number of pods that succeeded. For details, see Success policy.

• ServiceAccountTokenNodeBinding

  ServiceAccountTokenNodeBinding moved to Beta in Kubernetes 1.31. You can create a service account token that is directly bound to a node. The token defines the node information and verifies whether the node is available. The token will be valid until it expires or either the associated node is deleted. For details, see Manually create an API token for a ServiceAccount.

Kubernetes 1.30

• Webhook matching expression

  The Webhook matching expression feature moved to GA. This feature enables admission webhooks to be matched based on specific conditions, providing control over the triggering conditions of the webhooks in a more precise granularity. For details, see Dynamic Admission Control.

• Validating admission policies

  Validating admission policies moved to GA. This feature allows you to declare the validating admission policies of resources using Common Expression Language (CEL). For details, see Validating Admission Policy.

• Horizontal pod auto scaling based on container resource metrics

  The horizontal pod auto scaling feature based on container resource metrics advanced to GA. This feature allows HPA to configure auto scaling based on the resource usage of each container within a pod, rather than just the overall resource usage of the pod. This makes it easier to set scaling thresholds for the most critical containers in a pod. For details, see Container resource metrics.

• Legacy ServiceAccount token cleaner

  The legacy ServiceAccount token cleaner moved to GA. It runs as part of kube-controller-manager and checks every 24 hours to see if any auto-generated legacy ServiceAccount token has not been used in a specific amount of time (one year by default, specified by --legacy-service-account-token-clean-up-period). If so, the cleaner marks those tokens as invalid and adds the kubernetes.io/legacy-token-invalid-since, with the current date as the value. If an invalid token is not used for a specific period of time (one year by default, specified by --legacy-service-account-token-clean-up-period), the cleaner deletes it. For details, see Legacy ServiceAccount token cleaner.

Kubernetes 1.29

• Load balancer IP mode for Services

  The load balancer IP mode is a new alpha feature. Kubernetes 1.29 adds the ipMode field to the Services' status field for configuring traffic forwarding from Services within a cluster to pods. If ipMode is set to VIP, traffic to the load balancer will be redirected to the target node by kube-proxy. If it is set to Proxy, traffic delivered to a node will be sent to the load balancer and then redirected to the target node by the load balancer. This feature addresses the issue that the load balancer is not used to distribute traffic. For details, see Load Balancer IP Mode for Services.

• nftables proxy mode

  The nftables proxy mode is a new alpha feature. This feature allows kube-proxy to run in nftables mode. In this mode, kube-proxy configures packet forwarding rules using the nftables API of the kernel netfilter subsystem. For details, see nftables proxy mode.

• Garbage collection for unused container images

  The garbage collection for unused container images is a new alpha feature. This feature allows you to specify the maximum time a local image can be unused for each node. If the time expires, the image will be garbage collected. To configure the setting, specify the ImageMaximumGCAge field for kubelet. For details, see Garbage collection for unused container images.

• PodLifecycleSleepAction

  PodLifecycleSleepAction is a new alpha feature. This feature introduces the sleep hook to the container lifecycle hooks. You can pause a container for a specified duration after it starts or before it is stopped by enabling this feature. For details, see Hook handler implementations.

• KubeletSeparateDiskGC

  KubeletSeparateDiskGC is a new alpha feature. With this feature enabled, container images and containers can be garbage collected even if they are on separate file systems. For details, see Feature Gates.

• ClusterTrustBundle projected volumes

  clusterTrustBundle projected volumes are new alpha features. With this feature enabled, the clusterTrustBundle projected volume source injects the contents of one or more ClusterTrustBundle objects as an automatically-updating file. For details, see clusterTrustBundle projected volumes.

• Image pull per runtime class

  Image pull per runtime class is a new alpha feature. With this feature enabled, the kubelet references container images by a tuple (of image name or runtime handler) rather than just the image name or digest. Your container runtime may adapt its behavior based on the selected runtime handler. Pulling images based on runtime classes will be helpful for VM based containers. For details, see Image pull per runtime class.

• PodReadyToStartContainers condition

  The PodReadyToStartContainers moved to beta. Kubernetes 1.29 introduces the PodReadyToStartContainers condition to the pods' status field. If it is set to true, the sandbox of a pod is ready and service containers can be created. This feature enables cluster administrators to gain a clearer and more comprehensive view of pod sandbox creation completion and container readiness. This enhanced visibility allows them to make better-informed decisions and troubleshoot issues more effectively. For details, see PodReadyToStartContainersCondition Moved to Beta.

• Job-related features
  • Pod replacement policy

    The pod replacement policy feature moved to beta. This feature ensures that a pod is replaced only when it reaches the Failed state, which means that status.phase becomes Failed. It does not recreate a pod when the deletion timestamp is not empty and the pod is still being deleted. This prevents two pods from occupying index and node resources concurrently.

  • Backoff limit per index

    The backoff limit per index moved to beta. By default, pod failures for indexed jobs are counted and restricted by the global limit of retries, specified by .spec.backoffLimit. This means that if there is a consistently failing index in a job, pods specified by the job will be restarted repeatedly until pod failures exhaust the limit. Once the limit is reached, the job is marked failed and pods for other indexes in the job may never be even started. The feature allows you to complete execution of all indexes, despite some indexes failing, and to better use the compute resources by avoiding unnecessary retries of consistently failing indexes.

  For details, see Jobs.

• Native sidecar containers

  Native sidecar containers moved to beta. The restartPolicy field is added to initContainers. When this field is set to Always, the sidecar container is enabled. The sidecar container and service container are deployed in the same pod. This cannot prolong the pod lifecycle. Sidecar containers are commonly used in scenarios such as network proxy and log collection. For details, see Sidecar Containers.

• The legacy ServiceAccount token cleaner

  Legacy ServiceAccount token cleaner moved to beta. It runs as part of kube-controller-manager and checks every 24 hours to see if any auto-generated legacy ServiceAccount token has not been used in a specific amount of time (one year by default, specified by --legacy-service-account-token-clean-up-period). If so, the cleaner marks those tokens as invalid and adds the kubernetes.io/legacy-token-invalid-since, with the current date as the value. If an invalid token is not used for a specific period of time (one year by default, specified by --legacy-service-account-token-clean-up-period), the cleaner deletes it. For details, see Legacy ServiceAccount token cleaner.

• DevicePluginCDIDevices

  DevicePluginCDIDevices moved to beta. With this feature enabled, plugin developers can use the CDIDevices field added to DeviceRunContainerOptions to pass CDI device names directly to CDI enabled runtimes. For details, see Device Plugins.

• PodHostIPs

  The PodHostIPs feature moved to beta. With this feature enabled, Kubernetes adds the hostIPs field to Status of pods and downward API to expose node IP addresses to workloads. This field specifies the dual-stack protocol version of the host IP address. The first IP address is always the same as the host IP address. For details, see Feature Gates (removed).

• API priority and fairness (APF)

  APF moved to GA. APF classifies and isolates requests in a more fine-grained way. It improves max-inflight limitations. It also introduces a limited amount of queuing, so that the API server does not reject any request in cases of very brief bursts. Requests are dispatched from queues using a fair queuing technique so that, for example, a poorly-behaved controller does not cause others (even at the same priority level) to become abnormal. For details, see API Priority and Fairness.

• APIListChunking

  APIListChunking moved to GA. This feature allows clients to perform pagination in List requests to avoid performance problems caused by returning too much data at a time. For details, see Feature Gates.

• lastPhaseTransitionTime of PersistentVolume (PV)

  lastPhaseTransitionTime moved to beta. With this feature enabled, Kubernetes adds the lastPhaseTransitionTime field to the status field of a PV to indicate the time when the PV phase changes last time. Cluster administrators are now able to track the last time a PV transitioned to a different phase, allowing for more efficient and informed resource management. For details, see PersistentVolume Last Phase Transition Time in Kubernetes.

• ReadWriteOncePod

  ReadWriteOncePod moved to GA. With this feature enabled, you can set the access mode to ReadWriteOncePod in a PersistentVolumeClaim (PVC) to ensure that only one pod can modify data in the volume at a time. This can prevent data conflicts or damage. For details, see ReadWriteOncePod.

• CSINodeExpandSecret

  CSINodeExpandSecret moved to GA. This feature allows secret authentication data to be passed to a CSI driver for use when a node is added. For details, see Volumes.

• CEL-based CustomResourceDefinition (CRD) verification

  The CEL-based CRD verification capability moved to GA. With this feature enabled, you are allowed to use the Common Expression Language (CEL) to define validation rules in CRDs, which are more efficient than webhook. For details, see CRD verification rules.

API Changes and Removals

Kubernetes 1.31

• In Kubernetes 1.31, the kubectl exec [POD] [COMMAND] command cannot be executed without a -- separator. In this case, you need to run kubectl exec [POD] -- [COMMAND].
• In Kubernetes 1.31, if caBundle is not empty but the value is invalid or it does not define any CA certificate, the CRD does not provide services. If caBundle is set to a valid value, it remains unchanged if updated. Attempting direct updates results in an "invalid field value" error, ensuring uninterrupted CRD services.

Kubernetes 1.30

• kubectl replaces prune-whitelist with prune-allowlist in the apply command.
• SecurityContextDeny, which has been deprecated in Kubernetes 1.27, is replaced by Pod Security Admission.

Kubernetes 1.29

• The time zone of a newly created CronJob cannot be configured using TZ or CRON_TZ in .spec.schedule. Use .spec.timeZone instead. CronJobs that have been created are not affected by this change.
• The alpha API ClusterCIDR is removed.
• The startup parameter --authentication-config is added to kube-apiserver to specify the address of the AuthenticationConfiguration file. This startup parameter is mutually exclusive with the --oidc-* startup parameter.
• The API version kubescheduler.config.k8s.io/v1beta3 of KubeSchedulerConfiguration is removed. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1.
• The CEL expressions are added to v1alpha1 AuthenticationConfiguration.
• ServiceCIDR is added. It allows you to specify a CIDR block for a ClusterIP Service.
• The startup parameters --conntrack-udp-timeout and --conntrack-udp-timeout-stream are added to kube-proxy. They are options for configuring the kernel parameters nf_conntrack_udp_timeout and nf_conntrack_udp_timeout_stream.
• CEL expressions are supported by WebhookMatchCondition of v1alpha1 AuthenticationConfiguration.
• The type of PVC.spec.Resource is changed from ResourceRequirements to VolumeResourceRequirements.
• onPodConditions in PodFailurePolicyRule is marked as optional.
• The API version flowcontrol.apiserver.k8s.io/v1beta3 of FlowSchema and PriorityLevelConfiguration has been upgraded to flowcontrol.apiserver.k8s.io/v1, and the following changes have been made:
  • PriorityLevelConfiguration: The .spec.limited.nominalConcurrencyShares field defaults to 30 if the field is omitted. To ensure compatibility with 1.28 API servers, specifying an explicit 0 is not allowed in the v1 version in 1.29. In 1.30, explicit 0 will be allowed in this field in the v1 API. The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in 1.32.
• The kube-proxy command line document is updated. kube-proxy does not bind any socket to the IP address specified by --bind-address.
• If CSI-Node-Driver is not running, NodeStageVolume calls will be retried.
• ValidatingAdmissionPolicy type checking now supports CRDs. To use this feature, the ValidatingAdmissionPolicy feature gate must be enabled.
• The startup parameter --nf-conntrack-tcp-be-liberal is added to kube-proxy. You can configure it by setting the kernel parameter nf_conntrack_tcp_be_liberal.
• The startup parameter --init-only is added to kube-proxy. Setting the flag makes kube-proxy init container run in the privileged mode, perform its initial configuration, and then exit.
• The fileSystem field of container is added to the response body of CRI. It specifies the file system usage of a container. Originally, the fileSystem field contains only the file system of the container images.
• All built-in cloud providers are disabled by default. If you still need to use them, you can configure the DisableCloudProviders and DisableKubeletCloudCredentialProvider feature gates to disable or enable cloud providers.

References

For more details about the performance comparison and function evolution between Kubernetes 1.31 and other versions, see the following documents:

• Kubernetes v1.31 Release Notes
• Kubernetes v1.30 Release Notes
• Kubernetes v1.29 Release Notes