ALM-19007 HBase Merge Queue Exceeds the Threshold (for 2.x and Earlier Versions)

Description

The system checks the HBase compaction queue size every 30 seconds. This alarm is generated when the compaction queue size exceeds the alarm threshold (100 by default) for three consecutive times. This alarm is cleared when the compaction queue size is less than the threshold.

Attribute

Parameters

Impact on the System

The cluster performance may deteriorate, affecting data read and write.

Possible Causes

• The number of HBase RegionServers is too small.
• There are too many regions on a RegionServer of HBase.
• The HBase RegionServer heap size is small.
• Resources are insufficient.
• Related parameters are not configured properly.

Procedure

1. Check whether related HBase parameters are properly configured.
   1. Log in to the MRS cluster details page, choose Components > HBase > Service Configuration, switch Basic Configuration to All Configurations, and search for hbase.hstore.compaction.min and hbase.hstore.compaction.max, and increase the values of hbase.regionserver.thread.compaction.small and hbase.regionserver.thread.compaction.throttle.
      

      If you did not synchronize IAM users, perform synchronization first. (In the Dashboard tab, click Synchronize next to IAM User Sync.)

   2. Save the configuration, and restart the HBase service during off-peak hours or perform a rolling restart to make the configuration take effect.
   3. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.

2. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact the O&M engineers and send the collected logs.

Alarm Clearing

This alarm is automatically cleared after the fault is rectified.

Related Information

None