ALM-13002 ZooKeeper Direct Memory Usage Exceeds the Threshold

Description

The system checks the direct memory usage of the ZooKeeper service every 30 seconds. This alarm is generated when the usage exceeds the threshold.

When the Trigger Count is 1, this alarm is cleared when the ZooKeeper Direct memory usage is less than the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the ZooKeeper Direct memory usage is less than 80% of the threshold.

Attribute

Parameters

Impact on the System

If the available memory of ZooKeeper is insufficient, memory overflow may occur and services may break down. As a result, upstream services (such as HDFS and Yarn) fail to run.

Possible Causes

The direct memory of the ZooKeeper instance is overused or the direct memory is inappropriately allocated.

Procedure

Check the direct memory usage.

1. Log in to MRS Manager, choose O&M > Alarm > Alarms, view the alarm details, and record the IP address of the instance for which the alarm is generated.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On Manager, choose Cluster > Services > ZooKeeper > Instances > quorumpeer (the instance for which the alarm is generated). Click the drop-down menu in the upper right corner of Chart, choose Customize > CPU and Memory, and select ZooKeeper Heap And Direct Buffer Resource Percentage, click OK.
3. Check whether the used direct memory of ZooKeeper reaches 80% of the maximum direct memory specified for ZooKeeper.
   • If yes, go to Step 4.
   • If no, go to Step 8.

4. On Manager, choose Cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer(Role) > System. Check whether -XX:MaxDirectMemorySize exists in the GC_OPTS parameter.
   • If yes, go to Step 5.
   • If no, go to Step 6.

5. Delete -XX:MaxDirectMemorySize from the value of the GC_OPTS parameter, save the configuration, and restart the ZooKeeper service.
   

   The service will be unavailable during the restart. In addition, upper-layer services that depend on the service are affected.

6. Check whether the ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold exists.
   • If yes, handle the alarm by referring to ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold.
   • If no, go to Step 7.

7. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 8.

Collect fault information.

8. On MRS Manager, choose O&M > Log > Download.
9. Select ZooKeeper in the required cluster from the Service.
10. Click the edit icon in the upper right corner, and select a time span starting 10 minutes before and ending 10 minutes after when the alarm was generated. Then, click Download to collect the logs.
11. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None