Configuring Node Fault Detection Policies

Prerequisites

The CCE Node Problem Detector add-on (CCE Node Problem Detector) add-on has been installed in the cluster.

Enabling Node Fault Detection

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Nodes. On the displayed page, click the Nodes tab.
3. Click Fault Detection Policies to check the current fault detection items. For more details, see CCE Node Problem Detector Check Items. If Fault Detection Policies is unavailable, the CCE Node Problem Detector add-on is not installed or running. Install or troubleshoot the add-on.
4. Check the node status for any abnormal metrics.

   

5. Click Abnormal metrics and rectify the fault as prompted.

   

Custom Check Items

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Nodes and click the Nodes tab. Then, click Fault Detection Policies.
3. On the displayed page, view the current check items. Click Edit in the Operation column and edit checks.
   Currently, the following configurations are supported:

   • Enable/Disable: Enable or disable a check item.
   • Target Node: By default, check items run on all nodes. Use node labels to filter target nodes. When multiple criteria are specified, all must match. If no criterion is set, all nodes are selected by default. For example, the spot ECS interruption check runs only on spot pricing nodes, filtered by label cce.io/is-spot.

     

   • Check Period: The interval between check executions. CCE Node Problem Detector provides a default value that covers common fault scenarios. You can change the value as needed.
   • Trigger: The CCE Node Problem Detector add-on provides the default threshold to match common fault scenarios. You can change the threshold as required. The threshold varies depending on check items, such as the number of failures and resource usage percentage. You can adjust the threshold as required. For example, you can change the threshold of resource usage percentage from 90% to 80%.

     

   • Policy: After a fault occurs, you can select the policies listed in the following table as needed.

     Table 1 Troubleshooting policies

     Policy	Effect
     Report exception	Kubernetes events are reported.
     Disable scheduling	Kubernetes events are reported and the NoSchedule taint is added to the node.
     Evict workload	Kubernetes events are reported and the NoExecute taint is added to the node. This operation will evict workloads on the node and interrupt services. Exercise caution when performing this operation.

CCE Node Problem Detector Check Items

Check items are supported only in the add-on 1.16.0 and later versions.

Check items cover events and statuses.

• Event-related

  For event-related check items, when a problem occurs, CCE Node Problem Detector reports an event to the API server. The event type can be Normal (normal event) or Warning (abnormal event).

  Table 2 Event-related check items

  Check Item	Function	Description
  OOMKilling	Listen to the kernel logs and check whether there are any OOM events. If there is an OOM event, the component will report it. Typical scenario: The memory used by the process in the container exceeds the limit, triggering OOM and terminating the process.	Warning event Listening object: /dev/kmsg Matching rule: "Killed process \\d+ (.+) total-vm:\\d+kB, anon-rss:\\d+kB, file-rss:\\d+kB.*"
  TaskHung	Listen to the kernel logs and check whether there are any taskHung events. If there is a taskHung event, the component will report it. Typical scenario: Disk I/O suspension causes process suspension.	Warning event Listening object: /dev/kmsg Matching rule: "task \\S+:\\w+ blocked for more than \\w+ seconds\\."
  ReadonlyFilesystem	Listen to the kernel logs and check whether there is a Remount root filesystem read-only error in the system kernel. Typical scenario: A user detaches a data disk from a node by mistake on the ECS, and applications continuously write data to the mount point of the data disk. As a result, an I/O error occurs in the kernel and the disk is remounted as a read-only disk. NOTE: If a node's rootfs uses Device Mapper and the data disk is detached from the node, the thin pool will malfunction. This will affect CCE Node Problem Detector, and the add-on will not be able to detect node faults.	Warning event Listening object: /dev/kmsg Matching rule: Remounting filesystem read-only

• Status-related

  For status-related check items, when a problem occurs, CCE Node Problem Detector reports an event to the API server and changes the node status synchronously. This function can be used together with Node-problem-controller fault isolation to isolate nodes.

  Table 3 Checking system components

  Check Item	Function	Description
  Container network component error CNIProblem	Check the status of the CNI components (container network components).	None
  Container runtime component error CRIProblem	Check the status of Docker and containerd of the CRI components (container runtime components).	Check object: Docker or containerd
  Frequent restarts of kubelet FrequentKubeletRestart	Periodically review system logs to check whether kubelet restarts frequently.	Default threshold: 4 restarts within 4 minutes If kubelet restarts 4 times within 4 minutes, a fault alarm will be generated. Listening object: logs in the /run/log/journal directory NOTE: Ubuntu and Huawei Cloud EulerOS 2.0 do not support these check items due to incompatible log formats.
  Frequent restarts of Docker FrequentDockerRestart	Periodically review system logs to check whether Docker restarts frequently.
  Frequent restarts of containerd FrequentContainerdRestart	Periodically review system logs to check whether containerd restarts frequently.
  kubelet error KubeletProblem	Check the status of kubelet.	None
  kube-proxy error KubeProxyProblem	Check the status of kube-proxy.	None
  PodIdentityAgent error PodIdentityAgentProblem	Check the status of PodIdentityAgent. This check item is valid for the add-on 1.19.50 or later. NOTE: If the cluster version does not support pod identities (available in v1.28.15-r80, v1.29.15-r40, v1.30.14-r40, v1.31.14-r0, v1.32.9-r0, v1.33.7-r0, v1.34.2-r0, and later versions), the alarm configuration for PodIdentityAgent errors in CCE Node Problem Detector will not take effect.	PodIdentityAgent on the node is used to obtain temporary credentials for accessing cloud services (such as EVS) based on the configured agency of the add-on or workload. If PodIdentityAgent is abnormal, access to the cloud services will be abnormal. This may affect services of the entire cluster. In this case, you need to evict the workloads on the node.

  Table 4 Checking system metrics

  Check Item	Function	Description
  Conntrack table full ConntrackFullProblem	Check whether the conntrack table is full.	Default threshold: 90% Usage: nf_conntrack_count Maximum value: nf_conntrack_max
  Insufficient disk resources DiskProblem	Check the usage of the system disk and CCE data disks (including the CRI logical disk and kubelet logical disk) on nodes.	Default threshold: 90% Source: df -h Data disks other than system and CCE data disks (including CRI and kubelet logical disks) cannot currently be checked on nodes.
  Insufficient file handles FDProblem	Check if the FD file handles are used up.	Default threshold: 90% Usage: the first value in /proc/sys/fs/file-nr Maximum value: the third value in /proc/sys/fs/file-nr
  Insufficient node memory MemoryProblem	Check whether memory is used up.	Default threshold: 80% Usage: MemTotal-MemAvailable in /proc/meminfo Maximum value: MemTotal in /proc/meminfo
  Insufficient process resources PIDProblem	Check whether PID process resources are exhausted.	Default threshold: 90% Usage: denominator of the fourth value in /proc/loadavg, which indicates the total number of processes that can run Maximum value: smaller value between /proc/sys/kernel/pid_max and /proc/sys/kernel/threads-max.

  Table 5 Checking the storage

  Check Item	Function	Description
  Disk read-only DiskReadonly	Periodically perform write tests on the system disk and CCE data disks (including the CRI logical disk and kubelet logical disk) on nodes to check the availability of key disks.	Detection paths: /mnt/paas/kubernetes/kubelet/ /var/lib/docker/ /var/lib/containerd/ /var/paas/sys/log/kubernetes The temporary file npd-disk-write-ping is generated in the detection path. Other data disks except the system and CCE data disks, including the CRI and kubelet logical disk, on nodes cannot be checked currently.
  emptyDir storage pool error EmptyDirVolumeGroupStatusError	Check whether the ephemeral volume groups on nodes are normal. Impact: Pods that depend on the storage pool cannot write data to the temporary volume. The temporary volume is remounted as a read-only file system by the kernel due to an I/O error. Typical scenario: When creating a node, a user configures two data disks as an ephemeral volume storage pool. Some data disks are deleted by mistake. As a result, the storage pool becomes abnormal.	Check period: 30s Source: vgs -o vg_name, vg_attr Principle: Check whether the VG (storage pool) is in the P state. If yes, some PVs (data disks) are lost. Joint scheduling: The scheduler can automatically identify a PV storage pool error and prevent pods that depend on the storage pool from being scheduled to the node. Exceptional scenario: CCE Node Problem Detector cannot detect VG (storage pool) loss caused by the loss of all PVs (data disks). In this case, kubelet automatically isolates the node, detects the loss of VGs (storage pools), and updates the corresponding resources in nodestatus.allocatable to 0. This prevents pods that depend on the storage pool from being scheduled to the node. Damage to a single PV cannot be detected by this check item, but it can be detected by the ReadonlyFilesystem check item.
  PV storage pool error LocalPvVolumeGroupStatusError	Check the PV groups on nodes. Impact: Pods that depend on the storage pool cannot write data to the persistent volume. The persistent volume is remounted as a read-only file system by the kernel due to an I/O error. Typical scenario: When creating a node, a user configures two data disks as a persistent volume storage pool. Some data disks are deleted by mistake.
  Mount point error MountPointProblem	Check the mount points on nodes. Definition: You cannot access a mount point by running the cd command. Typical scenario: Network File System (NFS), for example, obsfs and s3fs, is mounted to a node. When the connection is abnormal due to network or peer NFS server exceptions, all processes that access the mount point are suspended. For example, during a cluster upgrade, the kubelet is restarted, and all mount points are scanned. If the abnormal mount point is detected, the upgrade fails.	Alternatively, you can run the following command: for dir in `df -h | grep -v "Mounted on" | awk "{print \\$NF}"`;do cd $dir; done && echo "ok"
  Suspended disk I/O DiskHung	Check whether I/O suspension occurs on all nodes' disks, that is, whether I/O read and write operations are not responded. Definition of I/O suspension: The system does not respond to disk I/O requests, and some processes are in the D state. Typical scenario: Disks cannot respond due to abnormal OS hard disk drivers or severe faults on the underlying network.	Check object: all data disks Source: /proc/diskstat Alternatively, you can run the following command: iostat -xmt 1 Thresholds: (All following conditions must be met.) Average usage (ioutil) ≥ 0.99 Average I/O queue length (avgqu-sz) ≥ 1 Average I/O transfer volume ≤ 1 Average I/O transfer volume = Number of writes completed per second (iops, unit: w/s) + Amount of data written per second (ioth, unit: wMB/s) NOTE: In some OSs, no data changes during I/O suspension. In this case, calculate the CPU I/O time usage (iowait > 0.8).
  Slow disk I/O DiskSlow	Check whether all nodes' disks have slow I/Os, that is, whether I/Os respond slowly. Typical scenario: EVS disks have slow I/Os due to network fluctuation.	Check object: all data disks Source: /proc/diskstat Alternatively, you can run the following command: iostat -xmt 1 Default threshold: Average I/O latency (await) ≥ 5000 ms NOTE: This check item is invalid during I/O suspension. It is because CCE does not respond to any I/O requests and await is not updated.

  Table 6 Other check items

  Check Item	Function	Description
  Abnormal NTP NTPProblem	Check whether the node clock synchronization service ntpd or chronyd is running properly and whether there is a system time drift.	Default clock offset threshold: 8000 ms
  Process D error ProcessD	Check whether there is any process in the D state on nodes.	Default threshold: 10 abnormal processes detected for three consecutive times Source: /proc/{PID}/stat Alternately, you can run the ps aux command. Exceptional scenario: The ProcessD check item ignores the resident D processes (heartbeat and update) on which the SDI drivers on BMS nodes depend.
  Process Z error ProcessZ	Check whether there is any process in the Z state on nodes.
  RDMA network interface error	Check the RDMA network interface status. NOTE: CCE Node Problem Detector 1.19.37 and later versions support the RDMA network interface error detection function and mark the RDMAProblem status on the node. It automatically writes this status to the node object. If the add-on is rolled back to an earlier version that does not support this function, CCE Node Problem Detector cannot clear the status. As a result, the marked status is retained.	Default threshold: one RDMA network interface error detected for one consecutive time Source: Command: rdma link show
  ResolvConf error ResolvConfFileProblem	Check whether the ResolvConf file is lost. Check whether the ResolvConf file is normal. Definition: No upstream domain name resolution server (nameserver) is included.	Check object: /etc/resolv.conf
  Existing scheduled event ScheduledEvent	Check whether there is any live migration event on nodes. A live migration event is usually triggered by a hardware fault and is an automatic fault rectification method at the IaaS layer. Typical scenario: The host is faulty. For example, the fan is damaged or the disk has bad sectors. As a result, a live migration is triggered for VMs.	Source: http://169.254.169.254/meta-data/latest/events/scheduled This check item is an Alpha feature and is disabled by default.
  The spot price node is being reclaimed. SpotPriceNodeReclaimNotification	Check whether any spot price node is interrupted and reclaimed due to preemption.	Default check interval: 120 seconds Default fault handling policy: Evict some workloads on the nodes.

  The kubelet component has the following default check items, which have bugs or defects. You can fix them by upgrading the cluster or using CCE Node Problem Detector.

  Table 7 Default kubelet check items

  Check Item	Function	Description
  Insufficient PIDs PIDPressure	Check whether PIDs are sufficient.	Interval: 10 seconds Threshold: 90% Defect: In community version 1.23.1 and earlier, this check item becomes invalid when over 65,535 PIDs are used. For details, see issue 107107. In community version 1.24 and earlier, thread-max is not considered in this check item.
  Insufficient memory MemoryPressure	Check whether the allocatable memory for the containers is sufficient.	Interval: 10 seconds Threshold: Maximum value – 100 MiB Allocatable memory = Total memory on a node – Reserved memory on a node Defect: This check item checks only the allocatable memory of containers and does not check that on the node.
  Insufficient disk space DiskPressure	Check the disk usage and inode usage of the kubelet and Docker disks.	Interval: 10 seconds Threshold: 90%