ALM-50404 Slow Job Query on JobServer

Alarm Description

The system checks the latency of the job query API in JobServer every 30 seconds. By default, a major alarm is reported when the latency exceeds 10s for three consecutive times and a critical alarm is reported when the latency exceeds 20s for three consecutive times. The alarm is cleared when the latency is lower than the threshold for three consecutive times.

Alarm Attributes

Alarm Parameters

Impact on the System

It takes a long time to query a job, for example, through the REST API.

Possible Causes

The number of concurrent requests to JobServer on a node surges, or underlying services like DBService and YARN take a long time to respond.

Handling Procedure

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, search for the alarm "ALM-50404 Slow Job Query on JobServer", locate the abnormal JobServer instance node according to alarm information, and obtain CurrentValue in Additional Information.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On the alarm page, check whether there is a JobServer, DBService, or YARN fault alarm.
   • If yes, go to Step 3.
   • If no, go to Step 4.

3. Rectify the fault by referring to the alarm help. Then, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Collect fault information.

4. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
5. Expand the Service drop-down list, and select JobGateway for the target cluster.
6. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.

7. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None