ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold
Description
The system checks the heap memory usage of the ZooKeeper service every 60 seconds. The alarm is generated when the heap memory usage of a ZooKeeper instance exceeds the threshold (95% of the maximum memory).
The alarm is cleared when the memory usage is less than the threshold.
Attribute
Alarm ID |
Alarm Severity |
Auto Clear |
---|---|---|
13004 |
Major |
Yes |
Parameters
Name |
Meaning |
---|---|
Source |
Specifies the cluster for which the alarm is generated. |
ServiceName |
Specifies the service name for which the alarm is generated. |
RoleName |
Specifies the role name for which the alarm is generated. |
HostName |
Specifies the object (host ID) for which the alarm is generated. |
Trigger Condition |
Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated. |
Impact on the System
If the available ZooKeeper heap memory is insufficient, a memory overflow occurs and the service breaks down.
Possible Causes
The heap memory of the ZooKeeper instance is overused or the heap memory is inappropriately allocated.
Procedure
Check heap memory usage.
- On the FusionInsight Manager portal, On the displayed interface, click the drop-down button of ZooKeeper Heap Memory Usage Exceeds the Threshold and confirm the node IP address of the host for which the alarm is generated in the Location Information.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Instance, click quorumpeer in the Role column of the corresponding IP address. Click the drop-down menu in the upper right corner of Chart, choose Customize > CPU and Memory, and select ZooKeeper Heap And Direct Buffer Resource Percentage, click OK. Check the heap memory usage.
- Check whether the used heap memory of ZooKeeper reaches 95% of the maximum heap memory specified for ZooKeeper.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer > System. Increase the value of -Xmx in GC_OPTS as required. The details are as follows:
- On the Instance tab, click quorumpeer in the Role column of the corresponding IP address. Choose Customize > CPU and Memory in the upper right corner, and select ZooKeeper Heap And Direct Buffer Resource, click OK to check the heap memory used by ZooKeeper.
- Change the value of -Xmx in the GC_OPTS parameter based on the actual heap memory usage. Generally, the value is twice the size of the ZooKeeper data volume. For example, if 2 GB ZooKeeper heap memory is used, the following configurations are recommended: -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=512M -XX:MetaspaceSize=64M -XX:MaxMetaspaceSize=64M -XX:CMSFullGCsBeforeCompaction=1
- Save the configuration and restart the ZooKeeper service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Related Information
None
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.