ALM-18022 Insufficient Yarn Queue Resources

Alarm Description

• The alarm module checks Yarn queue resources every 60 seconds. This alarm is generated when available resources or ApplicationMaster (AM) resources of a queue are insufficient.

  This alarm is cleared when available resources are sufficient.

Alarm Attributes

Alarm Parameters

Impact on the System

• It takes long time to end an application.
• A new application cannot run after submission.

Possible Causes

• NodeManager node resources are insufficient.
• The configured maximum resource capacity of the queue is excessively small.
• The configured maximum ApplicationMaster resource percent is too small.

Handling Procedure

Check NodeManager resources.

1. On the FusionInsight Manager, choose O&M > Alarm > Alarms.
2. View the Location in the details page of Insufficient Yarn Queue Resources. Check whether QueueName is root and QueueMetric is Memory, or QueueName is root and QueueMetric is vCores.
   • If yes, go to Step 3.
   • If no, go to Step 4.

3. If the location information is displayed, the memory or CPU of the Yarn cluster is insufficient. Log in to the NodeManager node and query the available memory and CPU of the node.

   Query the available memory of the node.

   free -g

   Query the available CPU of the node.

   cat /proc/cpuinfo

   On FusionInsight Manager, increase the values of the Yarn NodeManager resource parameters yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores based on the queried resources.

   Restart the NodeManager instance and check whether the alarm is cleared.

   

   During NodeManager restart, containers submitted to this node may be retried to other nodes.

   • If yes, no further action is required.
   • If no, go to Step 4.

Check the maximum resource capacity of a queue.

4. View location information of this alarm and check whether QueueName is <Tenant Queue> and QueueMetric is Memory, or QueueName is <Tenant Queue> and QueueMetric is vCores in Location, check whether available Memory = or available vCores = are included in Additional Information.
   • If yes, go to Step 5.
   • If no, go to Step 7.

5. The memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Resource Distribution Policy and increase the value of Max Resources. Then, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 6.

6. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

   If Additional Information contains available Memory =, change the value of yarn.queue.memory.alarm.threshold to a value smaller than that of available Memory = in Additional Information.

   If Additional Information contains available vCores =, change the value of yarn.queue.vcore.alarm.threshold to a value smaller than that of available vCores = in Additional Information.

   Check whether the alarm is cleared 5 minutes later.

   • If yes, no further action is required.
   • If no, go to Step 9.

Check the maximum ApplicationMaster resource percentage.

7. If available AmMemory = or available AmvCores = is included in Additional Information, ApplicationMaster memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Queue Configuration and increase the value of Maximum Am Resource Percent. Then, check whether this alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 8.

8. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

   If Additional Information contains available AmMemory =, change the value of yarn.am.memory.alarm.threshold to a value smaller than that of available AmMemory = in Additional Information.

   If Additional Information contains available AmvCores =, change the value of yarn.am.vcore.alarm.threshold to a value smaller than that of available AmvCores = in Additional Information.

   Check whether the alarm is cleared 5 minutes later.

   • If yes, no further action is required.
   • If no, go to Step 9.

Collect fault information.

9. On FusionInsight Manager of the active cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
10. Expand the Service drop-down list, and select Yarn for the target cluster.
11. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
12. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None