ALM-19032 Number of Tasks in the RegionServer RPC Write Queue Exceeds the Threshold

Alarm Description

The system checks the number of tasks waiting in the RPC write queue for the RegionServer instances of the HBase service every 30 seconds. This alarm is generated when the number of waiting tasks exceeds the threshold for 10 consecutive times.

This alarm is cleared when the number of waiting tasks is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

Request queues are stacked, and the RegionServer memory GC pressure increases. As a result, the response time of write requests increases. For latency-sensitive services, a large number of service write requests may time out.

Possible Causes

• The RegionServer heap memory is not properly configured.
• A slow disk fault occurred.
• The RegionServer configuration is improper.
• Regions of RegionServers are not evenly distributed and hotspotting occurred.
• The latency of WAL Sync operations is high.

Handling Procedure

1. Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, locate the row containing the alarm whose Alarm ID is 19032, and view the service instance and host name in Location.

Check RegionServer heap memory.

2. In the alarm list on FusionInsight Manager, check whether the "Heap Memory Usage of the HBase Process Exceeds the Threshold" alarm is generated for the service instance in 1.
   • If yes, go to 3.
   • If no, go to 5.

3. Rectify the fault by following the handling procedure of "ALM-19008 Heap Memory Usage of the HBase Process Exceeds the Threshold".
4. Wait several minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 5.

5. On FusionInsight Manager, choose Cluster > Services > HBase > Chart, select GC from the chart category, and check whether the GC times and GC monitoring period are normal.
   • If yes, go to 6.
   • If no, go to 9.

6. Click Configurations, search for GC_OPTS, and increase the value of Xmx of the RegionServer within the allowed memory range. Set the value to a number less than or equal to 31 GB. Click Save.
7. Click Dashboard and click More > Restart Service to restart the HBase service.
8. Wait several minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check for slow disk fault.

9. Check whether alarm "Slow Disk Fault" or "Disk Unavailable" are generated on the node.
   • If yes, go to 10.
   • If no, go to 12.

10. Rectify the fault by following the handling procedure of "ALM-12033 Slow Disk Fault" or "ALM-12063 Disk Unavailable".
11. Wait several minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

Check the RegionServer configuration.

12. On FusionInsight Manager, choose Cluster > Service > HBase, click Configurations > All Configurations, and check whether the values of hbase.wal.hsync and hbase.hfile.hsync are true.
    • If yes, go to 13.
    • If no, go to 15.

13. Set both hbase.wal.hsync and hbase.hfile.hsync to false and click Save. Click Dashboard and click More > Restart Service to restart the HBase service.
14. Wait several minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 15.

Check whether RegionServer regions are evenly distributed.

15. On FusionInsight Manager, choose Cluster > Services > HBase. Click HMaster(Active) on the right of HMaster Web UI to go to the web UI of the HBase instance. View the Base Stats tab in the Region Servers area. Check whether the number of regions in the Num.Regions column is even.
    • If yes, go to 20.
    • If no, go to 16.

    

16. Log in to the node where the HBase client is deployed as the omm user.
17. Run the following commands to go to the client installation directory and set the environment variable:

    cd Client installation directory

    source bigdata_env

    kinit the supergroup user group or a user with the Global Admin permission (If Kerberos authentication is disabled for the cluster, skip this operation.)

18. Run the following commands to enable HBase load balancing and check whether the function is successfully enabled:

    hbase shell

    balance_switch true

    balancer_enabled

    If the command output is true, load balancing is enabled.

    Run the balancer command to manually trigger the load balancing function.

    

    You are advised to enable and trigger load balancing during off-peak hours.

19. Wait several minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 20.

Check the WAL sync latency.

20. On FusionInsight Manager, choose Cluster > Services > HBase > Chart. In the Chart Category area, select Operations. Check whether the value of "P99.9th Percentile of WAL Sync Operation Delay-All Instances" exceeds 500 ms.
    • If yes, go to 21.
    • If no, go to 22.

21. Click Instances, select the RegionServer instance for which the alarm is generated, and choose More > Restart Instance. You also need to perform 22 and provide the logs to O&M engineers for fault locating.

Collect fault information.

22. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
23. Expand the Service drop-down list, and select HBase for the target cluster.
24. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
25. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.