ALM-18023 Number of Pending Yarn Tasks Exceeds the Threshold

Alarm Description

The alarm module checks the number of pending applications in the Yarn root queue every 60 seconds. The alarm is generated when the number exceeds 60.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
18023	Major	Quality of service	Yarn	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the cluster service for which the alarm was generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
	QueueName	Identifies the queue for which the alarm is generated.

Impact on the System

It takes long time to end an application.
A new application cannot run after submission.

Possible Causes

NodeManager node resources are insufficient.
The maximum resource capacity of the queue and the maximum AM resource percentage are too small.
The monitoring threshold is too small.

Handling Procedure

Check NodeManager resources.

On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager (Active) to access the ResourceManager web UI.
Click Scheduler and check whether the root queue resources are used up in Application Queues.
- If yes, go to 3.
- If no, go to 4.
Expand the capacity of the NodeManager instance of the Yarn service. After the capacity expansion, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.

Check the maximum queue resource capacity and the maximum AM resource percentage.

Check whether the resources of the queue corresponding to the pending task are used up.
- If yes, go to 5.
- If no, go to 6.
On FusionInsight Manager, choose Tenant Resources > Dynamic Resource Plan and add resources as required. Check whether the alarms are cleared.
- If yes, no further action is required.
- If no, go to 6.

Adjust the monitoring thresholds.

On FusionInsight Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > Applications > Pending Applications, and increase the thresholds as required.
Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 8.

Collect fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select Yarn for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.