ALM-13002 ZooKeeper Direct Memory Usage Exceeds the Threshold

Alarm Description

The system checks the direct memory usage of the ZooKeeper service every 30 seconds. The alarm is generated when the direct memory usage of a ZooKeeper instance exceeds the threshold.

When the Trigger Count is 1, this alarm is cleared when the ZooKeeper Direct memory usage is less than the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the ZooKeeper Direct memory usage is less than 80% of the threshold.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
13002	Critical (default threshold: 90%) Major (default threshold: 80%)	Quality of service	ZooKeeper	Yes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

13002

Critical (default threshold: 90%)

Major (default threshold: 80%)

Quality of service

ZooKeeper

Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service name for which the alarm is generated.
	RoleName	Specifies the role name for which the alarm is generated.
	HostName	Specifies the object (host ID) for which the alarm is generated.
Additional Information	Trigger Condition	Specifies the threshold for triggering the alarm.

Impact on the System

If the available memory of ZooKeeper is insufficient, memory overflow may occur and services may break down. As a result, upstream services (such as HDFS and Yarn) fail to run.

Possible Causes

The direct memory of the ZooKeeper instance is overused or the direct memory is inappropriately allocated.

Handling Procedure

Check the direct memory usage.

On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of ZooKeeper Direct Memory Usage Exceeds the Threshold. Check the IP address of the instance that reports the alarm.
On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Instance > quorumpeer(the IP address checked). Click the drop-down menu in the upper right corner of Chart, choose Customize > CPU and Memory, and select ZooKeeper Heap And Direct Buffer Resource Percentage, click OK.
Check whether the used direct buffer memory of ZooKeeper reaches 80% of the maximum direct buffer memory specified for ZooKeeper.
- If yes, go to 4.
- If no, go to 8.
On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer > System to check whether "-XX:MaxDirectMemorySize" exists in the GC_OPTS parameter.
- If yes, in the GC_OPTS parameter, delete "-XX:MaxDirectMemorySize" and go to 5.
- If no, go to 6.
Save the configuration and restart the ZooKeeper service.
Check whether the ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold exists.
- If yes, handle the alarm by referring to ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold.
- If no, go to 7.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.

Collect fault information.

On the FusionInsight Manager portal, choose O&M > Log > Download.
Select ZooKeeper in the required cluster from the Service.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.