ALM-18003 NodeManager Unhealthy

Description

The system checks the number of unhealthy NodeManager nodes every 30 seconds, and compares the number with the threshold. The Unhealthy Nodes indicator has a default threshold. This alarm is generated when the value of the Unhealthy Nodes indicator exceeds the threshold.

To change the threshold, choose Cluster > Services > Yarn > Configurations > All Configurations and change the value

of yarn.nodemanager.unhealthy.alarm.threshold. You do not need to restart Yarn to make the change take effect.

The default threshold is 0. The alarm is generated when the number of unhealthy nodes exceeds the threshold, and is cleared when the number of unhealthy nodes is less than the threshold.

Attribute

Parameters

Impact on the System

• The faulty NodeManager node cannot provide the Yarn service.
• The number of containers decreases, so the cluster performance deteriorates.

Possible Causes

• The hard disk space of the host where the NodeManager node resides is insufficient.
• User omm does not have the permission to access a local directory on the NodeManager node.

Procedure

Check the hard disk space of the host.

1. On MRS Manager, choose O&M > Alarm > Alarm, and click to view the alarm details, and obtain the unhealthy node in Additional Information.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Choose Cluster > Services > Yarn > Instances, click the NodeManager instance corresponding to the host, choose Instance Configuration > All Configurations, and search for and record the values of yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
   • yarn.nodemanager.local-dirs: Storage location of files after localization.
   • yarn.nodemanager.log-dirs: Storage location of container logs.

3. Choose O&M > Alarm > Alarms. In the alarm list, check whether the related disk has the alarm ALM-12017 Insufficient Disk Capacity.
   • If yes, go to Step 4.
   • If no, go to Step 5.

4. Rectify the fault by following the instructions provided in ALM-12017 Insufficient Disk Capacity. After the fault is rectified, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

5. Choose Hosts > Name of the desired host . On the Dashboard page, check the disk usage of the corresponding partition. Check whether the percentage of the used space of the mounted disk exceeds the value of yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

   yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage: Maximum disk usage (in percentage) allowed after a disk is marked as faulty. On Manager, choose Cluster > Services > Yarn > Configurations > All Configurations, and search for and view the value of this parameter.

   • If yes, go to Step 6.
   • If no, go to Step 7.

6. Reduce the disk usage to less than the configured value, wait for 10 to 20 minutes, and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Check the access permission of the local directory on each NodeManager node.

7. Log in to each NodeManager node as user root and go to the directory obtained in Step 2.
8. Run the ll command to check whether the permission of the localdir and containerlogs folders is 755 and whether User:Group is omm:ficommon.
   • If yes, go to Step 10.
   • If no, go to Step 9.

9. Run the following command to set the permission to 755 and User:Group to omm:ficommon:

   chmod 755 <folder_name>

   chown omm:ficommon <folder_name>

10. Wait for 10 to 20 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 11.

Collect fault information.

11. On MRS Manager of the cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
12. Select Yarn in the required cluster from the Service.
13. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
14. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None