HBase Application Development Suggestions

Avoiding Invoking the closeRegion Method of Admin to Close a Region

Admin provides an API for closing a region:

public void closeRegion(final String regionname, final String serverName)

When this method is used to close a region, the HBase client directly sends an RPC request to the RegionServer where the region is located. However, the entire process is unaware to the Master. That is, although the RegionServer closes the region, the Master still believes that it is opened on the RegionServer. In this case, if the Master determines to move the region during a balancing operation, it will fail because the region is unavailable for closing. This is a defect in the current HBase version.

Therefore, it is not recommended to use this method to close a region.

Writing Data Using PutList

The Table class in HBase provides two APIs for writing data:

1. public void put(final Put put) throws IOException
2. public void put(final List<Put> puts) throws IOException

The first method has obvious disadvantages in performance compared with the second method. Therefore, it is recommended to use the second method to write data to HBase.

Specifying StartKey and EndKey During Scan

A scan with a defined range can greatly improve performance.

Code example:

Scan scan = new Scan();
scan.addColumn(Bytes.toBytes("familyname"),Bytes.toBytes("columnname"));
scan.setStartRow(Bytes.toBytes("rowA")); // Assume that the StartKey is rowA.
scan.setStopRow(Bytes.toBytes("rowB")); // Assume that the EndKey is rowB.
for(Result result : demoTable.getScanner(scan)) {
// process Result instance
}

Keeping WAL Enabled

Write-Ahead-Log (WAL) is a mechanism that ensures data security by writing data into log files before the data is written into the database.

By default, the WAL is enabled. However, the Put class provides the following API for disabling the WAL:

public void setWriteToWAL(boolean write)

You are advised not to call this API to disable WAL (that is, setting writeToWAL to False), as this may cause data loss in the latest 1s (the time is determined by the hbase.regionserver.optionallogflushinterval parameter on the RegionServer, and the default value is 1s). That said, if a high write speed is required and data loss in the most recent second is acceptable, you can disable WAL.

Set blockcache to true When Creating a Table or Scanning

When creating a table or scanning data on the HBase client, set blockcache to true. This setting should be configured based on specific application requirements, particularly whether the data will be frequently queried. If there are many repeated queries, setting blockcache to true can improve efficiency. Otherwise, you are advised to leave it disabled.

By default, blockcache is set to true. To set it to false, run the following command:

HColumnDescriptor fieldADesc = new HColumnDescriptor("value".getBytes());
fieldADesc.setBlockCacheEnabled(false);

Supporting Only Storage and Scan by Rowkeys, No Conditional Queries or ORDER BY

HBase stores data based on Rowkeys in a lexicographical order. You are advised not to use HBase in random search and sorting scenarios.

Suggestions on Service Table Design

1. Pre-split regions to ensure even region distribution and improve concurrency.
2. Avoid excessive hotspot regions. You can introduce the time factor to the Rowkey based on application requirements.
3. Continuously store data that is accessed at the same time. Specifically, store the data that is read at the same time in adjacent rows, in the same row, or in the same cell.
4. Place frequently queried attributes in the front part of RowKeys. The sorting of RowKeys must match the main query conditions.
5. Select attributes with high dispersion for RowKeys. Analyze the data dispersion characteristics and query scenarios, and design the RowKeys based on the scenarios.
6. Store redundant information to improve retrieval performance. You can use secondary indexes to support a wider range of query scenarios.
7. Set the expiration time and number of versions so that expired data can be automatically deleted from the table.

In HBase, a region that is constantly writing data is called a hotspot region.

Suggestions on HBase Client Timeout Configuration

The timeout parameters related to the HBase client are as follows:

• hbase.rpc.timeout: indicates the timeout interval of an RPC operation, in milliseconds. The default value is 60000 (60 seconds). This parameter is used to control the maximum duration of a single RPC operation.
• hbase.client.scanner.timeout.period: indicates the timeout interval of a scanner operation, in milliseconds. The default value is 360000 (360 seconds). If no data is returned within this period, the operation times out.
• hbase.client.operation.timeout: indicates the timeout interval of a client operation, in milliseconds. The default value is 120000 (2 minutes). A complete client operation includes operations such as Get, Put, and Delete.
• hbase.client.retries.number: indicates the maximum number of retries for all retryable operations.

When configuring timeout parameters on the client, consider the following conditions to ensure that the total timeout covers the retry operations:

The value of hbase.client.operation.timeout must be larger than that of hbase.rpc.timeout multiplied by hbase.client.retries.number.

If the service has requirements on the request latency, you can adjust the preceding configurations. It is recommended that the value of hbase.rpc.timeout be no less than 5 seconds and the number of retries (hbase.client.retries.number) be no less than 5 to ensure service stability.

Pre-splitting HBase Tables

Differences between pre-split and non-pre-split HBase data tables

Core concepts of pre-splitting

                Pre-splitting boundaries
 Region 1        Region 2        Region 3
[  , "1a")      ["1a", "3c")    ["3c",   )
   ↓               ↓               ↓
RegionServer A  RegionServer B  RegionServer C

• Split key: defines the RowKey range of each region.
• Region count: determined by the number of split keys. Number of regions = Number of split keys + 1.
• Data routing: RowKeys are routed to the corresponding region based on the lexicographic order.

Pre-splitting strategies

• Pre-splitting based on HexStringSplit

  Application scenario: Suitable for RowKeys with hexadecimal hash prefixes (0-9, a-f).

  Table creation:

  Method 1: Manually specify hexadecimal split points.

  create 'my_table', 'cf', SPLITS => [
    '1', '2', '3', '4', '5', '6', '7', '8', '9',
    'a', 'b', 'c', 'd', 'e', 'f'
  ]

  Method 2: Use the built-in HBase shell splitter.

  create 'my_table', 'cf', {NUMREGIONS => 16, SPLITALGO => 'HexStringSplit'}

  Region distribution:

  Region	RowKey Range
  Region 1	[ , 1)
  Region 2	[1, 2)
  ...	...
  Region 16	[f, )

• Pre-splitting based on the service data volume

  Application scenario: Applicable to scenarios where the data volume can be estimated and more refined splitting control is required.

  Calculation procedure:

  1. Estimate the total data volume (number of rows and storage size).
  2. Determine the target size of a single region.
  3. Calculate the number of pre-split regions.

     Calculate the number of pre-splitting regions as follows: Number of regions = Estimated total data volume (GB)/Target size (GB) of a single region.

     When performing pre-splitting, consider the data growth trend.

     • Calculate the number of pre-splitting regions based on 1.5 to 2 times of the current data volume.
     • Start with a smaller region (5 to 7 GB) to reserve space for data growth.
     • Set a proper value for MAX_FILESIZE or set a proper region split size at the cluster level to control the automatic split threshold.
     • Adjust the split policy based on service requirements.

     Region splitting configuration:

     Parameter	Description	Default Value
     hbase.hregion.max.filesize	Maximum size of an HStoreFile, in bytes. If any HStoreFile in a column family exceeds the size, the managed HRegion will be split into two.	10G
     hbase.regionserver.region.split.policy	Defines the split policy. Currently, available split policies include ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy, DelimitedKeyPrefixRegionSplitPolicy, and KeyPrefixRegionSplitPolicy.	org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy

  4. Generate the region boundaries based on the hash prefix range.
• Pre-splitting based on the time range

  Application scenario: Suitable for scenarios where RowKeys contain time prefixes and the queries are primarily based on time ranges.

  Time prefixes can easily cause write hotspots (concentrated writes for current time data). Therefore, this method is used only when write volume is controllable or data is already scattered.

  Precautions for monthly pre-splitting

  • The data volume may vary greatly between different months, resulting in uneven region sizes.
  • Regularly evaluate whether to manually split large regions or merge small regions.
  Pre-splitting strategy selection

  Scenario	Recommended Strategy	Reason
  RowKeys with hexadecimal hash prefixes	HexStringSplit	Naturally aligns with hexadecimal prefixes, ensuring even data distribution.
  Estimable data volume	Estimated service data volume	Allows precise control over region sizes, avoiding overly small or large regions.
  Queries primarily by time	Time range-based pre-splitting	High scan efficiency, but requires careful management of write hotspots.
  Unpredictable data distribution	UniformSplit	Evenly splits byte space, making it a versatile choice.