Checking Slow and Oversized HBase Requests

Scenario

Query information about slow or oversized requests on the HBase shell CLI. Slow requests refer to the requests whose RPC response exceeds the threshold (value of hbase.ipc.warn.response.time set in the HBase service, 3000 ms by default) after you run the hbase shell command to query the server. Oversized requests refer to the requests whose data volume returned for each RPC exceeds the threshold (value of hbase.ipc.warn.response.size set in the HBase service, 5 MB by default) when you run the hbase shell command to query the server.

By default, each RegionServer node caches the latest 256 slow requests and oversized requests. You can adjust the cache size by configuring the hbase.regionserver.slowlog.ringbuffer.size parameter on the HBase service page in FusionInsight Manager.

This section applies to MRS 3.3.0 or later.

Command Description

This operation involves the following new HBase shell commands:

get_slowlog_responses: queries information about slow requests.
get_largelog_responses: queries information about oversized requests.
clear_slowlog_responses: clears data cached in RegionServers.

You can run the following command in the HBase shell to view how to use related commands:

help 'cmdName'

For example, run the help 'clear_slowlog_responses' command to view the usage of the clear_slowlog_responses command.

Click to enlarge

Viewing Request Information

The usage and supported parameters of get_slowlog_responses and get_largelog_responses are the same. This part describes how to use get_slowlog_responses.

You have logged in to the HBase shell CLI by referring to Using the HBase Client.

Run the following command to view the slow requests of all RegionServers:
get_slowlog_responses '*', {'LIMIT' => 50}

The LIMIT parameter specifies the number of records returned by each RegionServer. If this parameter is not specified, 10 records are returned by default.
Run the following command to view the slow requests of a specified RegionServer:
get_slowlog_responses ['SERVER_NAME1', 'SERVER_NAME2']

SERVER_NAME1 and SERVER_NAME2 indicate the server names of the RegionServers whose slow requests are to be viewed. To view the server names, log in to FusionInsight Manager, choose Cluster > Services > HBase, click the hyperlink next to HMaster WebUI to access the HBase web UI, obtain the server names of all RegionServers from ServerName on the Base Stats tab in the Region Servers area. One or more parameters are supported.
Run the following command to view the slow requests of a specified table and region:
get_slowlog_responses '*', {'TABLE_NAME' => 't1'}

get_slowlog_responses '*', {'REGION_NAME' => 'hbase:meta,,1'}

get_slowlog_responses '*', {'REGION_NAME' => 'hbase:meta,,1', 'TABLE_NAME' => 't1', 'FILTER_BY_OP' => 'AND'}

TABLE_NAME and REGION_NAME indicate the names of the specified table and region, respectively. If 'FILTER_BY_OP' => 'AND' is specified, each returned result must match all specified conditions. If the parameter is not specified, only any condition needs to be matched.
Run the following command to view the slow requests of a specified user and client:
get_slowlog_responses '*', {'USER' => 'user_name', 'CLIENT_IP' => '192.162.1.40:60225'}

USER and CLIENT_IP indicate the username, client IP address, and port number to be matched. If 'FILTER_BY_OP' => 'AND' is specified, the result that matches both USER and CLIENT_IP is returned. If the parameter is not specified, only USER or CLIENT_IP needs to be matched.