ALM-18022 Insufficient YARN Queue Resources

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The alarm module checks YARN queue resources periodically (controlled by the alarm.resource.lack.check.times.threshold parameter, in minutes). When the available queue resources or ApplicationMaster (AM) queue resources are insufficient:

If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is left blank, all queues are allowed to trigger this alarm.
If alarm.resource.lack.enable is set to true and alarm.resource.lack.enable.queues is set to a queue name, only the specified queue is allowed to report this alarm.
If alarm.resource.lack.enable is set to false, all queues are not allowed to report this alarm.

To set the preceding parameters, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations on FusionInsight Manager.

This alarm is cleared when available resources are sufficient.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
18022	Minor	Quality of service	Yarn	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	QueueName	Identifies the queue for which the alarm is generated.
	QueueMetric	Identifies the queue indicator for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the alarm triggering condition.

Impact on the System

It takes long time to end an application.
A new application cannot run after submission.

Possible Causes

Alarm reporting needs to be adjusted (applicable only to MRS 3.3.1 or later).
NodeManager node resources are insufficient.
The maximum resource capacity of the queue is too small.
The configured maximum ApplicationMaster resource percent is too small.

Handling Procedure

Adjusting the alarm reporting mechanism (applicable only to MRS 3.3.1 or later)

Check whether all queues need to report this alarm.
- If no queue needs to report alarms, log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable, change the value to false, and save the configuration.
- If only some queues need to report alarms: Log in to FusionInsight Manager, choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, search for alarm.resource.lack.enable.queues and change the value to the name of the queue for which this alarm needs to be reported, and save the configuration.
- If alarms need to be reported for all queues, go to 3.
Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 3.

Check NodeManager resources.

On FusionInsight Manager, choose O&M > Alarm > Alarms. The Alarms page is displayed.
View the Location in the details page of Insufficient YARN Queue Resources. Check whether QueueName is root and QueueMetric is Memory, or QueueName is root and QueueMetric is vCores
- If yes, go to 5.
- If no, go to 7.
The memory or CPU of the YARN cluster is insufficient. In this case, log in to the NodeManager node and run the free –g and cat /proc/cpuinfo commands to query the available memory and available CPU of the node, respectively. On the FusionInsight Manager, increase the values of yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores for the YARN NodeManager based on the query results. Then, restart the NodeManager instance.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.

Checking the maximum resource capacity of a queue

Check the Location of this alarm in the details page. Check whether QueueName is the queue name and QueueMetric is Memory, or QueueName is the queue name and QueueMetric is vCores. Then, check whether Additional Information in the alarm details contains available Memory = or available vCores =.
- If yes, go to 8.
- If no, go to 11.
The memory or CPU of the tenant queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Resource Distribution Policy and increase the value of Max Resources. Then check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

If Additional Information contains available Memory =, change the threshold of yarn.queue.memory.alarm.threshold to a value less than the available Memory = value.

If Additional Information contains available vCores =, change the threshold of yarn.queue.vcore.alarm.threshold to a value less than the available vCores = value.
Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 11.

Checking the maximum AM resource percentage

If available AmMemory = or available AmvCores = is included in Additional Information, ApplicationMaster memory or CPU of the queue is insufficient. In this case, choose Tenant Resources > Dynamic Resource Plan > Queue Configurations, and increase the value of Max Master Shares. Then, check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 12.
Choose Cluster > Services > Yarn. On the displayed page, click Configurations > All Configurations, enter the keyword threshold, click ResourceManager, and adjust the thresholds of the following parameters:

If Additional Information contains available AmMemory =, change the threshold of yarn.am.memory.alarm.threshold to a value less than the available AmMemory = value.

If Additional Information contains available AmvCores =, change the threshold of yarn.am.vcore.alarm.threshold to a value less than the available AmvCores = value.
Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 14.

Collect fault information.

On FusionInsight Manager of the active cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select Yarn for the target cluster.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.