ALM-19011 RegionServer Region Number Exceeds the Threshold

Alarm Description

The system checks the number of regions on each RegionServer in each HBase service instance every 30 seconds. The region number is displayed on the HBase service monitoring page and RegionServer role monitoring page. This alarm is generated when the number of regions on a RegionServer exceeds the threshold (default value: 2000) for 20 consecutive times. The threshold can be changed by choosing O&M > Alarm > Thresholds > HBase. This alarm is cleared when the number of regions is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

If the number of RegionServer regions exceeds the threshold, too many Regions increase the load of RegionServer, causing resource bottlenecks such as memory, disk I/O, and CPU. As a result, request response becomes slow or even times out.

Possible Causes

• The RegionServer region distribution is unbalanced.
• The HBase cluster scale is too small.

Handling Procedure

View alarm location information.

1. Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, locate the row containing the alarm whose Alarm ID is 19011, and view the service instance and host name in Location.
2. On the FusionInsight Manager homepage, choose Cluster > Services > HBase. Click HMaster(Active) to go to the WebUI of the HBase instance and check whether regions are evenly distributed on the RegionServer.
   

   By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.

   • If yes, go to Step 9.
   • If no, go to Step 3.
   Figure 1 WebUI of HBase instance
   

Enable load balancing.

3. Log in to the node where the HBase client is deployed as the root user.

   For details about how to log in to a cluster node, see Logging In to an MRS Cluster Node.

   Run the following command to open the client installation directory:

   cd Client installation directory

   Configure environment variables:

   source bigdata_env

   If the cluster uses the security mode, perform security authentication.

   kinit Service user who has the HBase operation permissions

   Enter the required password.

4. Run the following command to go to the HBase shell CLI:

   hbase shell

   Check whether the load balancing function is enabled:

   balancer_enabled

   • If yes, go to Step 6.
   • If no, go to Step 5.

5. In the HBase shell CLI, run the following command to enable the load balancing function:

   balance_switch true

   Verify that the function is enabled:

   balancer_enabled

6. Run the following command to manually trigger load balancing:

   balancer

   Enable and manually trigger load balancing during off-peak hours.

7. On the FusionInsight Manager homepage, choose Cluster > Services > HBase, and click HMaster(Active). On the displayed WebUI of the HBase instance, refresh the page and check whether regions are evenly distributed.
   • If yes, go to Step 8.
   • If no, go to Step 21.

8. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 9.

Delete unnecessary HBase tables.

9. On the FusionInsight Manager homepage, choose Cluster > Services > HBase and click HMaster(Active). On the displayed WebUI of the HBase instance, view tables stored in the HBase service instance and record unnecessary tables that can be deleted.
10. In the HBase shell CLI, run disable and drop to delete the unnecessary tables to reduce regions.

    disable 'Name of the table to be deleted'

    drop 'Name of the table to be deleted'

    

    Exercise caution to ensure data is deleted correctly.

11. In the HBase shell CLI, run the following command to check whether the load balancing function is enabled:

    balancer_enabled

    • If yes, go to Step 13.
    • If no, go to Step 12.

12. In the HBase shell CLI, run the following command to enable the load balancing function:

    balance_switch true

    Verify that the function is enabled:

    balancer_enabled

13. In the HBase shell CLI, run the following command to manually trigger the load balancing function:

    balancer

14. On the FusionInsight Manager homepage, choose Cluster > Services > HBase and click HMaster(Active). Click the HBase service instance that the alarm was generated for. On the displayed WebUI, refresh the page and check whether regions are evenly distributed.
    • If yes, go to Step 15.
    • If no, go to Step 21.

15. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 16.

Adjust the threshold.

16. On the FusionInsight Manager homepage, choose O&M > Alarm > Thresholds > HBase > Regions(RegionServer), select the applied rule, and click Modify to check whether the threshold is proper.
    • If it is excessively small, increase the threshold as required and go to Step 17.
    • If it is proper, go to Step 18.
    Figure 2 Regions(RegionServer_1)
    

17. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 18.

Perform system capacity expansion.

18. Add nodes to the HBase cluster and add RegionServer instances to the nodes. Then enable and manually trigger the load balancing function.
19. On the FusionInsight Manager homepage, choose Cluster > Services and click HMaster(Active). Click the HBase service instance that the alarm was generated for. On the displayed WebUI, refresh the page and check whether regions are evenly distributed.
    • If yes, go to Step 20.
    • If no, go to Step 21.

20. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 21.

Collect fault information.

21. On the FusionInsight Manager home page of the active and standby clusters, choose O&M > Log > Download.
22. Select HBase in the required cluster from the Service.
23. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
24. Contact O&M personnel and send the collected fault logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None