ALM-43020 Non-Heap Memory Usage of the IndexServer Process Exceeds the Threshold

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The system checks the non-heap memory usage of the IndexServer process every 30 seconds. A major alarm is reported if the usage is greater than 85% but less than 95% of the maximum memory. A critical alarm is reported if the usage is greater than 95% of the maximum memory.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
43020	Major (default threshold: 85% of the maximum memory) Critical (default threshold: 95% of the maximum memory.)	Quality of service	Spark	Yes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

43020

Major (default threshold: 85% of the maximum memory)

Critical (default threshold: 95% of the maximum memory.)

Quality of service

Spark

Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Identifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the alarm triggering condition.

Impact on the System

If the non-heap memory usage of the IndexServer process is too high, the performance deteriorates, and even memory overflow occurs. As a result, the IndexServer process is unavailable, and Carbon tasks with indexing enabled are slow or fail to run.

Possible Causes

The non-heap memory of the IndexServer process is overused or the non-heap memory is inappropriately allocated.

Handling Procedure

Check non-heap memory usage.

On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43020. Check the role name and the IP address of the host where the alarm is generated in Location.
On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the IndexServer that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > IndexServer Memory Usage Statistics > OK. Check whether the non-heap memory used by the process reaches the maximum non-heap memory threshold (95% by default).
- If the threshold is reached, go to 3.
- If the threshold is not reached, go to 7.
On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the IndexServer that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Statistics for the non-heap memory of the IndexServer process > OK. Based on the alarm generation time, check the values of the used non-heap memory of the process in the corresponding period and obtain the maximum value.
On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Configurations > All Configurations > IndexServer > Performance. You can change the value of XX:MaxMetaspaceSize in the spark.driver.extraJavaOptions parameter based on the ratio of the maximum non-heap memory used by the IndexServer process to the threshold specified by IndexServerNon-Heap Memory Usage Statistics (IndexServer) in the alarm period.

NOTE:

On FusionInsight Manager, you can choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark > Memory > IndexServer Non-Heap Memory Usage Statistics (IndexServer) to view the threshold.
Restart all IndexServer instances.
Check whether the alarm is cleared 10 minutes later.
- If the alarm is cleared, no further action is required.
- If the alarm is not cleared, go to 7.

Collect fault information.

On FusionInsight Manager, choose O&M > Log > Download.
Expand the Service drop-down list, and select Spark for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.