Help Center/ Cloud Container Engine/ User Guide/ Node Pools/ Managing Node Pools/ Node Self-Healing
Updated on 2026-09-21 GMT+08:00
View PDF
Share

Node Self-Healing

Node self-healing is an automatic node fault recovery capability provided by CCE. When a specified fault type occurs on a node and persists longer than a defined threshold, CCE automatically triggers a self-healing task that attempts to rectify the fault by restarting the node. This reduces manual intervention and improves cluster availability.

Prerequisites

Advanced self-healing requires the following:

  • A Kubernetes cluster meeting the following version requirements is available:
    • v1.30: v1.30.14-r100 or later
    • v1.31: v1.31.14-r60 or later
    • v1.32: v1.32.13-r30 or later
    • v1.33: v1.33.12-r10 or later
    • v1.34: v1.34.8-r10 or later
    • v1.35: v1.35.5-r10 or later
    • v1.36: v1.36.2-r0 or later
    • Clusters of later versions
  • CCE Node Problem Detector has been installed.

Precautions

  • Restarting a node evicts or interrupts all pods running on it. Ensure your workloads support HA deployment.
  • If the fault persists after the self-healing task is executed, CCE does not retry indefinitely. You must manually locate and resolve the fault.
  • If a node fails to self-heal, the node pool will not trigger another self-healing node restart until the fault is rectified.
  • In a node pool, only one node can be restarted for self-healing at a time.
  • After a node is restarted for self-healing, CCE will not trigger self-healing for that node again within 2 hours.
  • Bare metal nodes and HyperNodes do not support node self-healing.

How It Works

Self-healing is implemented by a daemon process and the systemd service manager. The overall process is as follows:

  1. Scheduled health check: The daemon process runs on the backend and periodically performs health checks on target components. Detection methods include process liveness detection and port response detection.
  2. Exception determination: If a component fails to respond or returns an error state for multiple consecutive checks, the daemon process flags it as abnormal. A single detection failure does not trigger self-healing, preventing false restarts caused by transient issues such as network jitter.
  3. Triggering self-healing: Once a component is flagged as abnormal, the daemon process invokes systemd to restart the service, enabling automatic recovery.

If Allow node restart when system and Kubernetes components are abnormal is enabled, CCE enters the node-level self-healing process if the fault persists after the component is restarted.

Self-Healing Process

After a self-healing task is triggered, CCE performs the following steps:

  1. Marks the affected node as rectifying: The node status is set to Rectifying, and the scheduler stops scheduling new pods to the node.
  2. Evicts pods on the node: An eviction taint is added to the node to trigger graceful eviction of existing pods. Pods configured to tolerate the taint are not affected. The timeout period is the greater of 10 minutes and the maximum TerminationGracePeriodSeconds of all pods to be evicted on the faulty node. The maximum timeout period is 30 minutes. If eviction fails after the timeout expires, subsequent operations continue regardless.
  3. Restarts the node: CCE restarts the node to attempt to restore the faulty component.
  4. Checks fault rectification: After the restart completes, CCE checks whether the fault is rectified.
    • Restoration successful: The eviction taint is removed and the node status is restored to normal to accept pod scheduling again.
    • Restoration failed: The node remains abnormal, and the self-healing task is marked as failed. Manual intervention is required.

Fault Types

Table 1 Fault types supported by basic self-healing

Fault Type

Description

Threshold

Self-Healing Action

CNI error

The CNI is malfunctioning, affecting pod creation.

50s

Restart the CNI.

kubelet error

The kubelet is unavailable or malfunctioning. The node status cannot be reported.

50s

Restart kubelet.

CRI error

The CRI is malfunctioning. Containers cannot be created or managed.

50s

Restart containerd or Docker.

kube-proxy error

kube-proxy is malfunctioning.

50s

Restart kube-proxy.

PodIdentityAgent error

The PodIdentityAgent is malfunctioning, which may affect CCE add-ons and services that depend on Pod Identity.

50s

Restart pod-identity-agent.
Table 2 Fault types supported by advanced self-healing

Fault Type

Description

Threshold

Self-Healing Action

CNI error

The CNI is malfunctioning, affecting pod creation.

50s

Restart the CNI.

kubelet error

The kubelet is unavailable or malfunctioning. The node status cannot be reported.

180s

1. Restart kubelet (triggered every 50s).

2. If Allow node restart when system and Kubernetes components are abnormal is enabled, restart the affected ECS.

CRI error

The CRI is malfunctioning. Containers cannot be created or managed.

180s

1. Restart containerd or Docker (triggered every 50s).

2. If Allow node restart when system and Kubernetes components are abnormal is enabled, restart the affected ECS.

kube-proxy error

kube-proxy is malfunctioning.

50s

Restart kube-proxy.

PodIdentityAgent error

The PodIdentityAgent is malfunctioning, which may affect CCE add-ons and services that depend on Pod Identity.

50s

Restart pod-identity-agent.

Filesystem read-only (CCE Node Problem Detector v1.19.75 or later)

The node filesystem is read-only and cannot be written. As a result, the kubelet and container runtime cannot work properly.

180s

If Allow node restart when system and Kubernetes components are abnormal is enabled, restart the affected ECS.

systemd offline

(CCE Node Problem Detector v1.19.75 or later)

systemd is malfunctioning. Containers cannot be started or destroyed.

180s

If Allow node restart when system and Kubernetes components are abnormal is enabled, restart the affected ECS.

kubeletNotReady (PLEG)

The PLEG health check fails.

180s

If Allow node restart when system and Kubernetes components are abnormal is enabled, restart the affected ECS.

Helpful Links

  • For details about the constraints, installation, and components of CCE Node Problem Detector, see CCE Node Problem Detector.

Common Issues

Restart Failure

  • Example Failure Messages 
    auto repair for kubelet failed: reboot job xxx failed: instance not found
auto repair for container runtime (containerd/docker) failed: reboot job xxx failed: internal server error
  • Cause Analysis

    CCE delivered a node restart command, but the restart operation failed due to an ECS error (such as abnormal instance status or underlying hardware faults).

  • Handling Procedure
    1. Log in to the ECS console, locate the target node, manually restart the node, and check whether the node is restored.
    2. If the fault persists, underlying resources may be faulty. In this case, submit a service ticket to contact technical support.

Fault Not Rectified After Node Restart

  • Example Failure Messages 
    auto repair for read-only filesystem failed: node not recovered in 15m0s
auto repair for kubelet PLEG failed: node not recovered in 15m0s
  • Cause Analysis

    If the fault persists after the node restart, the fault is not caused by a temporary exception. Instead, a more fundamental system fault (such as disk damage, kernel exception, or system file damage) exists and cannot be rectified by simply restarting the node.

  • Handling Procedure
    1. Reset the faulty node to reinstall the node OS to its initial state and add the node to the cluster again. For details, see Resetting a Node.
      • Ensure that there is no stateful data that cannot be migrated on the node, or data has been backed up.
      • Resetting a node will destroy all pods on the node. Ensure that your workloads support HA deployment or are ready for eviction.
    2. After the reset is complete, verify that the node is properly added to the cluster and that pods can be properly scheduled and run.
    3. If the fault persists after the reset, the underlying hardware may be irreversibly damaged. In this case, submit a service ticket to contact technical support.

Residual node.cce.io/repair Taint After Node Self-Healing

  • Symptom

    The node status is restored to Running, but the node.cce.io/repair taint still exists. As a result, new pods cannot be scheduled to the node.

  • Cause Analysis

    After the node is restarted, the node status is restored to available, but some components are not ready. The system retains the taint to prevent services from being scheduled to the node that is not ready.

  • Handling Procedure

    Check whether all components on the node are restored. If all components are restored, manually remove the taint to restore node scheduling. If components are not restored, submit a service ticket to contact technical support.

Parent Topic: Managing Node Pools