ALM-50405 Slow Job Stop on JobServer

Alarm Description

The system checks the failure rate of the job stop API in JobServer every 30 seconds. By default, a major alarm is reported when the failure rate exceeds 80% for three consecutive times, and a critical alarm is reported when it exceeds 90% for three consecutive times. The alarm is cleared when the failure rate falls below the threshold for three consecutive times.

Alarm Attributes

Alarm Parameters

Impact on the System

It takes a long time to stop a job, for example, through the REST API.

Possible Causes

The number of concurrent requests to JobServer on a node surges, or underlying services like YARN and DBService take a long time to respond.

Handling Procedure

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, search for the alarm "ALM-50405 Slow Job Stop on JobServer", locate the abnormal JobServer instance node according to alarm information, and obtain CurrentValue in Additional Information.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On the alarm page, check whether there is a JobServer, YARN, or DBService fault alarm.
   • If yes, go to Step 3.
   • If no, go to Step 4.

3. Rectify the fault by referring to the alarm help. Then, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Collect fault information.

4. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
5. Expand the Service drop-down list, and select JobGateway for the target cluster.
6. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.

7. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None