ALM-18022 Insufficient YARN Queue Resources

Alarm Description

The alarm module checks YARN queue resources periodically (controlled by the alarm.resource.lack.check.times.threshold parameter, in minutes). When the available queue resources or ApplicationMaster (AM) queue resources are insufficient:

• If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is left blank, all queues are allowed to trigger this alarm.
• If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is set to a queue name, only the specified queue is allowed to report this alarm.
• If alarm.resource.lack.enable is set to false, all queues are not allowed to report this alarm.

To set the preceding parameters, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations on FusionInsight Manager.

This alarm is cleared when available resources are sufficient.

Alarm Attributes

Alarm Parameters

Impact on the System

• It takes long time to end an application.
• A new application cannot run after submission.

Possible Causes

• Alarm reporting needs to be adjusted (applicable only to MRS 3.3.1 or later).
• NodeManager node resources are insufficient.
• The maximum resource capacity of the queue is too small.
• The configured maximum ApplicationMaster resource percent is too small.

Handling Procedure

Adjusting the alarm reporting mechanism (applicable only to MRS 3.3.1 or later)

1. Check whether all queues need to report this alarm.
   • If no queue needs to report alarms, log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable, change the value to false, and save the configuration.
   • If only some queues need to report alarms: Log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable.queues and change the value to the name of the queue for which this alarm needs to be reported, and save the configuration.
   • If alarms need to be reported for all queues, go to 3.

2. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to 3.

Check NodeManager resources.

3. On FusionInsight Manager, choose O&M > Alarm > Alarms. The Alarms page is displayed.
4. View the Location in the details page of Insufficient YARN Queue Resources. Check whether QueueName is root and QueueMetric is Memory, or QueueName is root and QueueMetric is vCores
   • If yes, go to 5.
   • If no, go to 7.

5. The memory or CPU of the YARN cluster is insufficient. In this case, log in to the NodeManager node and run the free –g and cat /proc/cpuinfo commands to query the available memory and available CPU of the node, respectively. On the FusionInsight Manager, increase the values of yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores for the YARN NodeManager based on the query results. Then, restart the NodeManager instance.
6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Checking the maximum resource capacity of a queue

7. Check the Location of this alarm in the details page. Check whether QueueName is the queue name and QueueMetric is Memory, or QueueName is the queue name and QueueMetric is vCores. Then, check whether Additional Information in the alarm details contains available Memory = or available vCores =.
   • If yes, go to 8.
   • If no, go to 11.

8. The memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Resource Distribution Policy and increase the value of Max Resources. Then check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

9. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

   If Additional Information contains available Memory =, change the threshold of yarn.queue.memory.alarm.threshold to a value less than the available Memory = value.

   If Additional Information contains available vCores =, change the threshold of yarn.queue.vcore.alarm.threshold to a value less than the available vCores = value.

10. Check whether the alarm is cleared 5 minutes later.
    • If yes, no further action is required.
    • If no, go to 11.

Checking the maximum AM resource percentage

11. If available AmMemory = or available AmvCores = is included in Additional Information, ApplicationMaster memory or CPU of the queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Queue Configurations, and increase the value of Max Master Shares. Then, check whether this alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

12. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

    If Additional Information contains available AmMemory =, change the threshold of yarn.am.memory.alarm.threshold to a value less than the available AmMemory = value.

    If Additional Information contains available AmvCores =, change the threshold of yarn.am.vcore.alarm.threshold to a value less than the available AmvCores = value.

13. Check whether the alarm is cleared 5 minutes later.
    • If yes, no further action is required.
    • If no, go to 14.

Collect fault information.

14. On FusionInsight Manager of the active cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
15. Expand the Service drop-down list, and select Yarn for the target cluster.
16. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
17. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.