Configuring HBase Table-Level Overload Control

Scenario

When the HBase requests soar in a short period of time, the system is overloaded. As a result, the P99 latency of requests increases, which severely affects services that relay on quick responses. HBase table-level overload prevention is used to control request latency of core tables (core services).

This section applies only to MRS 3.3.1 and later.

Procedure

1. Modify the properties of core tables and set table-level priorities.
   1. Log in to the node where the HBase client is installed as the client installation user and configure environment variables.

      cd HBase client installation directory

      source bigdata_env

   2. If Kerberos authentication is enabled for the cluster (the cluster is in security mode), run the following command to authenticate the user:

      kinit Component service user

   3. Run the following commands to log in to the HBase client and modify the table description:

      hbase shell

      alter 'test_table', PRIORITY=>'1'

      

      • The table priority can be set through the PRIORITY property. When the value of PRIORITY is greater than or equal to 1, the priority is high. You are advised to set PRIORITY to 1.
      • You can specify the PRIORITY property when creating a core table. For example:

        create 'test_table','cf',PRIORITY=>'1'

2. Log in to FusionInsight Manager, choose Cluster > Services > HBase > Configurations > All Configurations, search for the parameters listed in Table 1, and change the parameter values.

   Table 1 Parameters for HBase table-level overload control

   Parameter	Description	Suggestion
   hbase.ipc.server.default.callqueue.size.overload.threshold	RegionServer queue threshold. When the percentage of the request size in the queue exceeds the threshold, requests to low-priority tables are discarded. This configuration is used to control the request latency of core tables. When regular request restriction is enabled for protecting internal high-priority requests (the value of hbase.ipc.server.max.default.callqueue.size.ratio is not 0), the queue overload threshold is limited by the maximum percentage of regular requests in the queue, the final effective value is the product of the two values.	The number of core table requests and latency requirements must be considered. A smaller value is recommended. Generally, the value ranges from 0.5 to 1.0. The two types of overload control can be enabled independently or at the same time. RegionServer queue overload control is used when there is a large number of requests. RegionServer handler overload control is used when there is a large number of concurrent requests.
   hbase.ipc.server.handler.overload.threshold	RegionServer handler threshold. When the percentage of active handlers exceeds the threshold, requests to low-priority tables are discarded. This configuration is used to control the request latency of core tables.

3. Click Save. In the displayed dialog box, click OK.
4. Click Instances, select all RegionServer instances, and choose More > Restart Instance.