ALM-18025 Number of Terminated Yarn Tasks Exceeds the Threshold

Alarm Description

The alarm module checks the number of terminated applications in the Yarn root queue every 60 seconds. The alarm is generated when the number exceeds 50 for three consecutive times.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
18025	Major	Quality of service	Yarn	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the source in the cluster for which the alarm was generated.
	ServiceName	Specifies the cluster service for which the alarm was generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

A large number of application tasks are forcibly terminated.

Possible Causes

The user forcibly terminates a large number of tasks.
The system terminates tasks due to some error.

Handling Procedure

Check the alarm details.

On the FusionInsight Manager portal, choose O&M > Alarm > Alarms to go to the alarm page.
View Additional Information in the alarm details to check whether the alarm threshold is too small.
- If yes, go to 3.
- If no, go to 4.
Choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > Other > Terminated Applications of root queue to modify the threshold. Go to 6.
Choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active) to access the ResourceManager web UI.
Click KILLED in Applications and click the task on the top. View the description of Diagnostics and rectify the fault based on the task termination details (for example, the task is terminated by a user).
Wait for 3 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.

Collect the fault information.

On the FusionInsight Manager, choose O&M > Log > Download.
Expand the Service drop-down list, and select Yarn for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.