Help Center/ Cloud Container Engine/ User Guide/ Nodes/ Node O&M/ Node Component Self-Healing
Updated on 2026-08-13 GMT+08:00
View PDF
Share

Node Component Self-Healing

Overview

Node component self-healing is a core capability of CCE clusters that ensures high availability. When a key component on a node becomes unavailable due to an intermittent fault or abnormal exit, the self-healing mechanism automatically detects and restores the component without manual intervention. This minimizes the impact of node-level faults on workloads and maintains system availability.

How It Works

Self-healing is implemented by a daemon process and the systemd service manager. The overall process is as follows.

  1. Periodic health check

    The daemon process runs in the backend and periodically performs health checks on target components. Detection methods include:

    • Process liveness detection
    • Port response detection
  2. Exception judgment

    If a component fails to respond or returns an error status for multiple consecutive checks, the daemon process flags it as abnormal. A single detection failure does not trigger self-healing, preventing false restarts caused by transient issues such as network jitter.

  3. Self-healing trigger

    Once a component is flagged as abnormal, the daemon process invokes systemd to restart the service, enabling automatic recovery.

Self-Healing Components and Thresholds

Component

Detection Cycle

Consecutive Failure Threshold

Determination Duration

Self-Healing Behavior

yangtse/canal

10s

5

50s

Restart yangtse/canal.

containerd/Docker (container runtime)

10s

5

50s

Restart containerd or Docker.

kubelet

10s

5

50s

Restart kubelet.

kube-proxy

10s

5

50s

Restart kube-proxy.

pod-identity-agent

10s

5

50s

Restart pod-identity-agent.

A 10-second detection period × 5 consecutive failures = a 50-second determination window.

This window balances rapid fault detection with false-positive avoidance, ensuring sub-minute response time while filtering out transient jitter.

Auxiliary Detection by CCE Node Problem Detector

After CCE Node Problem Detector is installed, the cluster gains an additional layer of component status detection alongside self-healing, providing more comprehensive fault observability. The detection period and consecutive failure threshold can be configured per component to meet sensitivity and fault tolerance requirements in different scenarios. Shortening the period or lowering the threshold accelerates fault detection but may increase false positives. Conversely, increasing the period or raising the threshold improves fault tolerance but may delay fault response.

Component

Detection Cycle

Consecutive Failure Threshold

Determination Duration

Default Troubleshooting Policy

yangtse/canal

30s

1

30s

Report abnormal events.

containerd/Docker (container runtime)

30s

1

30s

Report abnormal events.

kubelet

30s

1

30s

Report abnormal events.

kube-proxy

30s

1

30s

Report abnormal events.

pod-identity-agent

30s

1

30s

Report abnormal events.

Advanced DR Policy

CCE Node Problem Detector can be configured to automatically perform the following disaster recovery (DR) actions after detecting a component fault, preventing the abnormal node from carrying further traffic.

Policy

Description

Effect

Mark as unschedulable

Adds the node.kubernetes.io/unschedulable taint to the affected node.

Prevents new pods from being scheduled to that node.

Evict pods

Triggers pod eviction on the node.

Migrates existing pods to healthy nodes, accelerating service recovery.

These policies can be enabled per component using CCE Node Problem Detector, making them suitable for production scenarios requiring high availability. Once the daemon process successfully restores the component, CCE Node Problem Detector updates the node status and removes the unschedulable taint.

Collaboration Between the Daemon Process and CCE Node Problem Detector

Category

Daemon Process Self-Healing

CCE Node Problem Detector

Positioning

Proactive recovery: automatically restarts components after detecting exceptions.

Automatic detection + DR: reports events and can be configured to mark nodes unschedulable or evict pods.

Sensitivity

50-second determination window (5 × 10 seconds), high fault tolerance.

Reports within 30 seconds (1 × 30 seconds); adjustable per component, yielding higher sensitivity.

Benefits

Automatic service recovery.

Rich fault event streams, supporting alarm-based troubleshooting and isolating faulty nodes to protect services.

Precautions

  • Self-healing applies only to component exceptions (such as process crashes or unresponsiveness). It cannot resolve system-level issues such as incorrect configurations, certificate expiration, or disk exhaustion.
  • During component restart, the corresponding node briefly enters the NotReady state. This is expected behavior.
  • Frequent self-healing triggers for a component indicate an underlying issue. Investigate using logs.
Parent Topic: Node O&M