ALM-18022 Insufficient Yarn Queue Resources

Alarm Description

• Versions earlier than MRS 3.3.1: The alarm module checks Yarn queue resources every 60 seconds. This alarm is generated when available resources or ApplicationMaster (AM) resources of a queue are insufficient.

  This alarm is cleared when available resources are sufficient.

• MRS 3.3.1 and later versions: The alarm module checks Yarn queue resources periodically (controlled by the alarm.resource.lack.check.times.threshold parameter, in minutes). When the available queue resources or ApplicationMaster (AM) queue resources are insufficient:
  • If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is left blank, all queues are allowed to trigger this alarm.
  • If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is set to a queue name, only the specified queue is allowed to report this alarm.
  • If alarm.resource.lack.enable is set to false, all queues are not allowed to report this alarm.

  To set the preceding parameters, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations on FusionInsight Manager.

  This alarm is cleared when available resources are sufficient.

Alarm Attributes

Alarm Parameters

Impact on the System

• It takes long time to end an application.
• A new application cannot run after submission.

Possible Causes

• Alarm reporting needs to be adjusted (applicable only to MRS 3.3.1 or later).
• NodeManager node resources are insufficient.
• The configured maximum resource capacity of the queue is excessively small.
• The configured maximum ApplicationMaster resource percent is too small.

Handling Procedure

Adjusting the alarm reporting mechanism (applicable only to MRS 3.3.1 or later)

1. Check whether all queues need to report this alarm.
   • If no queue needs to report alarms, log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable, change the value to false, and save the configuration.
   • If only some queues need to report alarms: Log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable.queues and change the value to the name of the queue for which this alarm needs to be reported, and save the configuration.
   • If alarms need to be reported for all queues, go to Step 3.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to Step 3.

Check NodeManager resources.

3. On the FusionInsight Manager, choose O&M > Alarm > Alarms.
4. View the Location in the details page of Insufficient Yarn Queue Resources. Check whether QueueName is root and QueueMetric is Memory, or QueueName is root and QueueMetric is vCores.
   • If yes, go to Step 5.
   • If no, go to Step 6.

5. If the location information is displayed, the memory or CPU of the Yarn cluster is insufficient. Log in to the NodeManager node and query the available memory and CPU of the node.

   For details about how to log in to a cluster node, see Logging In to an MRS Cluster Node.

   Query the available memory of the node.

   free -g

   Query the available CPU of the node.

   cat /proc/cpuinfo

   On FusionInsight Manager, increase the values of the Yarn NodeManager resource parameters yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores based on the queried resources.

   Restart the NodeManager instance and check whether the alarm is cleared.

   

   During NodeManager restart, containers submitted to this node may be retried to other nodes.

   • If yes, no further action is required.
   • If no, go to Step 6.

Check the maximum resource capacity of a queue.

6. View location information of this alarm and check whether QueueName is <Tenant Queue> and QueueMetric is Memory, or QueueName is <Tenant Queue> and QueueMetric is vCores in Location, check whether available Memory = or available vCores = are included in Additional Information.
   • If yes, go to Step 7.
   • If no, go to Step 9.

7. The memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Resource Distribution Policy and increase the value of Max Resources. Then, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 8.

8. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

   If Additional Information contains available Memory =, change the value of yarn.queue.memory.alarm.threshold to a value smaller than that of available Memory = in Additional Information.

   If Additional Information contains available vCores =, change the value of yarn.queue.vcore.alarm.threshold to a value smaller than that of available vCores = in Additional Information.

   Check whether the alarm is cleared 5 minutes later.

   • If yes, no further action is required.
   • If no, go to Step 11.

Check the maximum ApplicationMaster resource percentage.

9. If available AmMemory = or available AmvCores = is included in Additional Information, ApplicationMaster memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Queue Configuration and increase the value of Maximum Am Resource Percent. Then, check whether this alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 10.

10. Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

    If Additional Information contains available AmMemory =, change the value of yarn.am.memory.alarm.threshold to a value smaller than that of available AmMemory = in Additional Information.

    If Additional Information contains available AmvCores =, change the value of yarn.am.vcore.alarm.threshold to a value smaller than that of available AmvCores = in Additional Information.

    Check whether the alarm is cleared 5 minutes later.

    • If yes, no further action is required.
    • If no, go to Step 11.

Collect fault information.

11. On FusionInsight Manager of the active cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
12. Expand the Service drop-down list, and select Yarn for the target cluster.
13. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
14. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None