ALM-19021 Handler Usage of RegionServer Exceeds the Threshold

Updated on 2024-09-23 GMT+08:00

View PDF

Alarm Description

The system checks the RegionServer handler usage of each HBase service instance every 30 seconds. This alarm is generated when the handler usage of a RegionServer exceeds the threshold (90% for five consecutive times by default). This alarm is cleared if the handler usage is lower than or equal to the threshold.

Alarm Attributes

Alarm ID	Alarm Severity	Auto Cleared
19021	Major	Yes

Alarm Parameters

Parameter	Description
Source	Specifies the cluster for which the alarm was generated.
ServiceName	Specifies the service for which the alarm was generated.
RoleName	Specifies the role for which the alarm was generated.
HostName	Specifies the host for which the alarm was generated.
Trigger Condition	Specifies the threshold for triggering the alarm.

Impact on the System

RegionServer may not be able to provide services externally. As a result, the concurrent read/write processing become slow, or requests fail.

Possible Causes

The value of a handler is too small.
Hotspotting occurs.

Handling Procedure

View alarm location information.

Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms, locate the row that contains the alarm whose Alarm ID is 19021, and view the service instance and host name in Location.

Check the handler configuration.

Choose Cluster > Services > HBase and click the Configurations tab. In the upper right corner of the page, search for hbase.regionserver.handler.count and check whether its value is too small. The default value is 200.
- If yes, go to 3.
- If no, go to 5.
Change the value of this parameter to a larger value and save the configuration. Choose Cluster > Services > HBase, click the Instance tab, select the affected RegionServer instances, and choose More > Instance Rolling Restart. In the displayed dialog box, enter the username and password. In the Instance Rolling Restart dialog box, click OK and wait until the rolling restart is complete.
After the configuration takes effect, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.

Check whether cluster hotspotting occurs.

On FusionInsight Manager, choose Cluster > Services > HBase and click HMaster(Active) following HMaster WebUI to go to the web UI of the HBase instance. In the Region Servers area of the Home page, click Requests and check whether the requests in the Filtered Read Request Count and Write Request Count columns are evenly distributed.
- If yes, go to 13.
- If no, go to 6.
Check whether regions are evenly distributed.

On FusionInsight Manager, choose Cluster > Services > HBase and click HMaster(Active) following HMaster WebUI to go to the web UI of the HBase instance. In the Region Servers area of the Home page, click Base Stats and check whether the regions in the Num.Regions column are evenly distributed.
- If yes, go to 13.
- If no, go to 7.
Log in to the faulty RegionServer node as user omm.
Run the following commands to go to the client installation directory and set the environment variable:

cd Client installation directory

source bigdata_env

If the cluster uses the security mode, run the following command to perform security authentication:

kinit hbase

Enter the password as prompted (obtain the password from the MRS cluster administrator).
Run the following commands to check whether the load balancing function is enabled. If the command output is true, the load balancing function is enabled.

hbase shell

balancer_enabled
```
hbase:004:0> balancer_enabled
true
Took 0.0165 seconds
=> true
```
- If yes, go to 13.
- If no, go to 10.
Run the following commands in HBase Shell to enable the load balancing function and check whether the function is enabled.

balance_switch true

balancer_enabled

NOTE:

You are advised to enable and manually trigger the load balancing function during off-peak hours.
Run the following command to manually trigger' the load balancing function:

balancer
After the load balancing is complete, log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.

Collect the fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select HBase for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
Contact O&M personnel and provide the collected logs.