ALM-43019 Heap Memory Usage of the IndexServer2x Process Exceeds the Threshold

Description

The system checks the IndexServer2x process status every 30 seconds. The alarm is generated when the heap memory usage of a IndexServer2x process exceeds the threshold (95% of the maximum memory).

In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, IndexServer2x is changed to IndexServer. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

Attribute

Parameters

Impact on the System

If the heap memory usage of the IndexServer2x process is too high, the performance deteriorates, and even memory overflow occurs. As a result, the IndexServer2x process is unavailable, and Carbon tasks with indexing enabled are slow or fail to run.

Possible Causes

The heap memory of the IndexServer2x process is overused or the heap memory is inappropriately allocated.

Procedure

Check the heap memory usage.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43019. Check the role name and the IP address of the host where the alarm is generated in Location.
2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Instance. Click the IndexServer2x that reported the alarm to go to the Instance Status page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Memory > IndexServer2x Memory Usage Statistics > OK. Check whether the heap memory used by the IndexServer2x process has reached the upper limit (95% by default).
   • If the threshold is reached, go to Step 3.
   • If the threshold is not reached, go to Step 7.
   Figure 1 IndexServer2x memory usage statistics
   

3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Instance. Click the IndexServer2x that reported the alarm to go to the Instance Status page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Memory > Statistics for the heap memory of the IndexServer2x process > OK. Based on the alarm generation time, check the values of the used heap memory of the process in the corresponding period and obtain the maximum value.
   Figure 2 IndexServer2x heap memory statistics
   

4. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Configurations > All Configuration > IndexServer2x > Tuning. The default value of the SPARK_DRIVER_MEMORY parameter is 4 GB. You can change the value based on the ratio of the maximum heap memory used by the IndexServer2x process to the threshold specified by IndexServer2x Heap Memory Usage Statistics (IndexServer2x) in the alarm period. If the alarm persists after the parameter value is changed, increase the value by 0.5 times. If the alarm is generated frequently, increase the value by one time.
   

   On FusionInsight Manager, you can choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark2x > Memory > IndexServer2x Heap Memory Usage Statistics (IndexServer2x) to view the threshold.

5. Restart all IndexServer2x instances.
   

   When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.

6. Check whether the alarm is cleared in 10 minutes.
   • If the alarm is cleared, no further action is required.
   • If the alarm is not cleared, go to Step 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M > Log > Download.
8. Expand the Service drop-down list, and select Spark2x for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
10. Send the collected fault logs to O&M personnel for help.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Reference

None