Before You Start

v1.23 or v1.25

Upgraded to v1.27

Docker is no longer recommended. Use containerd instead. For details, see Container Engines.

This item has been included in the pre-upgrade check.

v1.21 or v1.19

Upgraded to v1.23

For the NGINX Ingress Controller of an earlier version (community version v0.49 or earlier, or CCE nginx-ingress version v1.x.x), the created ingresses can be managed by the NGINX Ingress Controller even if kubernetes.io/ingress.class: nginx is not set in the ingress annotations. However, for the NGINX Ingress Controller of a later version (community version v1.0.0 or later, or CCE nginx-ingress version v2.x.x), the ingresses created without specifying the Nginx type will not be managed by the NGINX Ingress Controller, and ingress rules will become invalid, which interrupts services.

This item has been included in the pre-upgrade check. You can also perform the self-check by referring to nginx-ingress.

v1.19 to v1.21

The bug of exec probe timeouts is fixed in Kubernetes 1.21. Before this bug is fixed, the exec probe does not consider the timeoutSeconds field. Instead, the probe will run indefinitely, even beyond its configured deadline. It will stop until the result is returned. If this field is not specified, the default value 1 is used. This field takes effect after the upgrade. If the probe runs over 1 second, the application health check may fail and the application may restart frequently.

Before the upgrade, check whether the timeout is properly set for the exec probe.

kube-apiserver of CCE v1.19 or later requires that the Subject Alternative Names (SANs) field be configured for the certificate of your webhook server. Otherwise, kube-apiserver fails to call the webhook server after the upgrade, and containers cannot be started properly.

Root cause: X.509 CommonName is discarded in Go v1.15. kube-apiserver of CCE v1.19 is compiled using Go v1.15. If your webhook certificate does not have SANs, kube-apiserver does not process the CommonName field of the X.509 certificate as the host name by default. As a result, the authentication fails.

Before the upgrade, check whether the SAN field is configured in the certificate of your webhook server.

• If you do not have your own webhook server, you can skip this check.
• If the field is not set, use the SAN field to specify the IP address and domain name supported by the certificate.

v1.15 to v1.19

The control plane of CCE clusters of v1.19 is incompatible with kubelet v1.15. If a node fails to be upgraded or the node to be upgraded restarts after the master node is successfully upgraded, there is a high probability that the node is in the NotReady status.

This is because the node failed to be upgraded restarts the kubelet and trigger the node registration. In clusters of v1.15, the default registration tags (failure-domain.beta.kubernetes.io/is-baremetal and kubernetes.io/availablezone) are regarded as invalid tags by the clusters of v1.19.

The valid tags in the clusters of v1.19 are node.kubernetes.io/baremetal and failure-domain.beta.kubernetes.io/zone.

1. In normal cases, this scenario is not triggered.
2. After the master node is upgraded, do not suspend the upgrade so the node can be quickly upgraded.
3. If a node fails to be upgraded and cannot be restored, evict applications on the node as soon as possible. Contact technical support and skip the node upgrade. After the upgrade is complete, reset the node.

In CCE v1.15 and v1.19 clusters, the Docker storage driver file system is switched from XFS to Ext4. As a result, the import package sequence in the pods of the upgraded Java application may be abnormal, causing pod exceptions.

Before the upgrade, check the Docker configuration file /etc/docker/daemon.json on the node. Check whether the value of dm.fs is xfs.

• If the value is ext4 or the storage driver is Overlay, you can skip the next steps.
• If the value is xfs, you are advised to deploy applications in the cluster of the new version in advance to test whether the applications are compatible with the new cluster version.

{
      "storage-driver": "devicemapper",
      "storage-opts": [
      "dm.thinpooldev=/dev/mapper/vgpaas-thinpool",
      "dm.use_deferred_removal=true",
      "dm.fs=xfs",
      "dm.use_deferred_deletion=true"
      ]
}

kube-apiserver of CCE v1.19 or later requires that the Subject Alternative Names (SANs) field be configured for the certificate of your webhook server. Otherwise, kube-apiserver fails to call the webhook server after the upgrade, and containers cannot be started properly.

Root cause: X.509 CommonName is discarded in Go v1.15. kube-apiserver of CCE v1.19 is compiled using Go v1.15. The CommonName field is processed as the host name. As a result, the authentication fails.

Before the upgrade, check whether the SAN field is configured in the certificate of your webhook server.

• If you do not have your own webhook server, you can skip this check.
• If the field is not set, use the SAN field to specify the IP address and domain name supported by the certificate.

In clusters of v1.17.17 and later, CCE automatically creates pod security policies (PSPs) for you, which restrict the creation of pods with unsafe configurations, for example, pods for which net.core.somaxconn under a sysctl is configured in the security context.

After an upgrade, you can allow insecure system configurations as required. For details, see Configuring a Pod Security Policy.

If initContainer or Istio is used in the in-place upgrade of a cluster of v1.15, pay attention to the following restrictions:

In kubelet v1.16 and later versions, QoS classes are different from those in earlier versions. In kubelet v1.15 and earlier versions, only containers in spec.containers are counted. In kubelet v1.16 and later versions, containers in both spec.containers and spec.initContainers are counted. The QoS class of a pod will change after the upgrade. As a result, the container in the pod restarts.

You are advised to modify the QoS class of the service container before the upgrade to avoid this problem. For details, see Table 1.

v1.13 to v1.15

After a VPC network cluster is upgraded, the master node occupies an extra CIDR block due to the upgrade of network components. If no container CIDR block is available for the new node, the pod scheduled to the node cannot run.

Generally, this problem occurs when the nodes in the cluster are about to fully occupy the container CIDR block. For example, the container CIDR block is 10.0.0.0/16, the number of available IP addresses is 65,536, and the VPC network allocates a CIDR block with the fixed size (using the mask to determine the maximum number of container IP addresses allocated to each node). If the upper limit is 128, the cluster supports a maximum of 512 (65536/128) nodes, including the three master nodes. After the cluster is upgraded, each of the three master nodes occupies one CIDR block. As a result, 506 nodes are supported.

Guaranteed

Besteffort

Burstable

Yes

Guaranteed

Burstable

Burstable

No

Guaranteed

Guaranteed

Guaranteed

No

Besteffort

Besteffort

Besteffort

No

Besteffort

Burstable

Burstable

No

Besteffort

Guaranteed

Burstable

Yes

Burstable

Besteffort

Burstable

Yes

Burstable

Burstable

Burstable

No

Burstable

Guaranteed

Burstable

Yes

FlowSchema and PriorityLevelConfiguration

flowcontrol.apiserver.k8s.io/v1beta2

flowcontrol.apiserver.k8s.io/v1

(This API has been available since v1.29.)

flowcontrol.apiserver.k8s.io/v1beta3

(This API has been available since v1.26.)

• Significant changes in flowcontrol.apiserver.k8s.io/v1:

  spec.limited.assuredConcurrencyShares of PriorityLevelConfiguration has been renamed spec.limited.nominalConcurrencyShares. The default value is 30 only when it is not specified, and the explicit value 0 does not change to 30.

• Key changes in flowcontrol.apiserver.k8s.io/v1beta3:

  spec.limited.assuredConcurrencyShares of PriorityLevelConfiguration has been renamed spec.limited.nominalConcurrencyShares.

CSIStorageCapacity

storage.k8s.io/v1beta1

storage.k8s.io/v1

(This API has been available since v1.24.)

None

FlowSchema and PriorityLevelConfiguration

flowcontrol.apiserver.k8s.io/v1beta1

flowcontrol.apiserver.k8s.io/v1beta3

(This API has been available since v1.26.)

None

HorizontalPodAutoscaler

autoscaling/v2beta2

autoscaling/v2

(This API has been available since v1.23.)

None

CronJob

batch/v1beta1

batch/v1

(This API has been available since v1.21.)

None

EndpointSlice

discovery.k8s.io/v1beta1

discovery.k8s.io/v1

(This API has been available since v1.21.)

Pay attention to the following changes:

• In each endpoint, the topology["kubernetes.io/hostname"] field has been deprecated. Replace it with the nodeName field.
• In each endpoint, the topology["kubernetes.io/zone"] field has been deprecated. Replace it with the zone field.
• The topology field is replaced with deprecatedTopology and cannot be written in v1.

Event

events.k8s.io/v1beta1

events.k8s.io/v1

(This API has been available since v1.19.)

Pay attention to the following changes:

• The type field can only be set to Normal or Warning.
• The involvedObject field is renamed regarding.
• The action, reason, reportingController, and reportingInstance fields are mandatory for creating a new events.k8s.io/v1 event.
• Use eventTime instead of the deprecated firstTimestamp field (this field has been renamed deprecatedFirstTimestamp and is not allowed to appear in the new events.k8s.io/v1 event object).
• Use series.lastObservedTime instead of the deprecated lastTimestamp field (this field has been renamed deprecatedLastTimestamp and is not allowed to appear in the new events.k8s.io/v1 event object).
• Use series.count instead of the deprecated count field (this field has been renamed deprecatedCount and is not allowed to appear in the new events.k8s.io/v1 event object).
• Use reportingController instead of the deprecated source.component field (this field has been renamed deprecatedSource.component and is not allowed to appear in the new events.k8s.io/v1 event object).
• Use reportingInstance instead of the deprecated source.host field (this field has been renamed deprecatedSource.host and is not allowed to appear in the new events.k8s.io/v1 event object).

HorizontalPodAutoscaler

autoscaling/v2beta1

autoscaling/v2

(This API has been available since v1.23.)

None

PodDisruptionBudget

policy/v1beta1

policy/v1

(This API has been available since v1.21.)

If spec.selector is set to null ({}) in PodDisruptionBudget of policy/v1, all pods in the namespace are selected. (In policy/v1beta1, an empty spec.selector means that no pod will be selected.) If spec.selector is not specified, pod will be selected in neither API version.

PodSecurityPolicy

policy/v1beta1

None

Since v1.25, the PodSecurityPolicy resource no longer provides APIs of the policy/v1beta1 version, and the PodSecurityPolicy access controller is deleted.

Use Pod Security Admission instead.

RuntimeClass

node.k8s.io/v1beta1

node.k8s.io/v1 (This API has been available since v1.20.)

None

MutatingWebhookConfiguration

ValidatingWebhookConfiguration

admissionregistration.k8s.io/v1beta1

admissionregistration.k8s.io/v1

(This API has been available since v1.16.)

• The default value of webhooks[*].failurePolicy is changed from Ignore to Fail in v1.
• The default value of webhooks[*].matchPolicy is changed from Exact to Equivalent in v1.
• The default value of webhooks[*].timeoutSeconds is changed from 30s to 10s in v1.
• The default value of webhooks[*].sideEffects is deleted, and this field must be specified. In v1, the value can only be None or NoneOnDryRun.
• The default value of webhooks[*].admissionReviewVersions is deleted. In v1, this field must be specified. (AdmissionReview v1 and v1beta1 are supported.)
• webhooks[*].name must be unique in the list of objects created through admissionregistration.k8s.io/v1.

CustomResourceDefinition

apiextensions.k8s.io/v1beta1

apiextensions/v1

(This API has been available since v1.16.)

• The default value of spec.scope is no longer Namespaced. This field must be explicitly specified.
• spec.version is deleted from v1. Use spec.versions instead.
• spec.validation is deleted from v1. Use spec.versions[*].schema instead.
• spec.subresources is deleted from v1. Use spec.versions[*].subresources instead.
• spec.additionalPrinterColumns is deleted from v1. Use spec.versions[*].additionalPrinterColumns instead.
• spec.conversion.webhookClientConfig is moved to spec.conversion.webhook.clientConfig in v1.
• spec.conversion.conversionReviewVersions is moved to spec.conversion.webhook.conversionReviewVersions in v1.

• spec.versions[*].schema.openAPIV3Schema becomes a mandatory field when the CustomResourceDefinition object of the v1 version is created, and its value must be a structural schema.
• spec.preserveUnknownFields: true cannot be specified when the CustomResourceDefinition object of the v1 version is created. This configuration must be specified using x-kubernetes-preserve-unknown-fields: true in the schema definition.
• In v1, the JSONPath field in the additionalPrinterColumns entry is renamed jsonPath (patch #66531).

APIService

apiregistration/v1beta1

apiregistration.k8s.io/v1

(This API has been available since v1.10.)

None

TokenReview

authentication.k8s.io/v1beta1

authentication.k8s.io/v1

(This API has been available since v1.6.)

None

LocalSubjectAccessReview

SelfSubjectAccessReview

SubjectAccessReview

SelfSubjectRulesReview

authorization.k8s.io/v1beta1

authorization.k8s.io/v1

(This API has been available since v1.16.)

spec.group was renamed spec.groups in v1 (patch #32709).

CertificateSigningRequest

certificates.k8s.io/v1beta1

certificates.k8s.io/v1

(This API has been available since v1.19.)

Lease

coordination.k8s.io/v1beta1

coordination.k8s.io/v1

(This API has been available since v1.14.)

None

Ingress

networking.k8s.io/v1beta1

extensions/v1beta1

networking.k8s.io/v1

(This API has been available since v1.19.)

• The spec.backend field is renamed spec.defaultBackend.
• The serviceName field of the backend is renamed service.name.
• The backend servicePort field represented by a number is renamed service.port.number.
• The backend servicePort field represented by a string is renamed service.port.name.
• The pathType field is mandatory for all paths to be specified. The options are Prefix, Exact, and ImplementationSpecific. To match the behavior of not defining the path type in v1beta1, use ImplementationSpecific.

IngressClass

networking.k8s.io/v1beta1

networking.k8s.io/v1

(This API has been available since v1.19.)

None

ClusterRole

ClusterRoleBinding

Role

RoleBinding

rbac.authorization.k8s.io/v1beta1

rbac.authorization.k8s.io/v1

(This API has been available since v1.8.)

None

PriorityClass

scheduling.k8s.io/v1beta1

scheduling.k8s.io/v1

(This API has been available since v1.14.)

None

CSIDriver

CSINode

StorageClass

VolumeAttachment

storage.k8s.io/v1beta1

storage.k8s.io/v1

• CSIDriver is available in storage.k8s.io/v1 since v1.19.
• CSINode is available in storage.k8s.io/v1 since v1.17.
• StorageClass is available in storage.k8s.io/v1 since v1.6.
• VolumeAttachment is available in storage.k8s.io/v1 since v1.13.

NetworkPolicy

extensions/v1beta1

networking.k8s.io/v1

(This API has been available since v1.8.)

None

DaemonSet

extensions/v1beta1

apps/v1beta2

apps/v1

(This API has been available since v1.9.)

• The spec.templateGeneration field is deleted.
• spec.selector is now a mandatory field and cannot be changed after the object is created. The label of an existing template can be used as a selector for seamless migration.
• The default value of spec.updateStrategy.type is changed to RollingUpdate (the default value in the extensions/v1beta1 API version is OnDelete).

Deployment

extensions/v1beta1

apps/v1beta1

apps/v1beta2

apps/v1

(This API has been available since v1.9.)

• The spec.rollbackTo field is deleted.
• spec.selector is now a mandatory field and cannot be changed after the Deployment is created. The label of an existing template can be used as a selector for seamless migration.
• The default value of spec.progressDeadlineSeconds is changed to 600 seconds (the default value in extensions/v1beta1 is unlimited).
• The default value of spec.revisionHistoryLimit is changed to 10. (In the apps/v1beta1 API version, the default value of this field is 2. In the extensions/v1beta1 API version, all historical records are retained by default.)
• The default values of maxSurge and maxUnavailable are changed to 25%. (In the extensions/v1beta1 API version, these fields default to 1.)

StatefulSet

apps/v1beta1

apps/v1beta2

apps/v1

(This API has been available since v1.9.)

• spec.selector is now a mandatory field and cannot be changed after the StatefulSet is created. The label of an existing template can be used as a selector for seamless migration.
• The default value of spec.updateStrategy.type is changed to RollingUpdate (the default value in the apps/v1beta1 API version is OnDelete).

ReplicaSet

extensions/v1beta1

apps/v1beta1

apps/v1beta2

apps/v1

(This API has been available since v1.9.)

spec.selector is now a mandatory field and cannot be changed after the object is created. The label of an existing template can be used as a selector for seamless migration.

PodSecurityPolicy

extensions/v1beta1

policy/v1beta1

(This API has been available since v1.10.)

PodSecurityPolicy for the policy/v1beta1 API version will be removed in v1.25.

etcd data backup

etcd data

Automatic backup during an upgrade

1-5 minutes

2 hours

Mandatory. The data is automatically backed up during an upgrade.

CBR cloud server backup

Master node disks, including component images, configurations, logs, and etcd data

One-click backup on a web page (manually triggered)

20 minutes to 2 hours (based on the cloud backup tasks in the current region)

20 minutes

This function is gradually replaced by EVS snapshot backup.

EVS snapshot backup

Master node disks, including component images, configurations, logs, and etcd data

One-click backup on a web page (manually triggered)

1-5 minutes

20 minutes

This function is coming soon.

After this function is released, it will replace CBR cloud server backup.