HBase Application Development Rules

Creating a Configuration Instance

This class should be instantiated by calling the create() method of HBaseConfiguration. Otherwise, the HBase configurations cannot be successfully loaded.

Correct example:

// This part should be declared in the declaration area of class member variables.
private Configuration hbaseConfig = null;
// It is recommended that this class be instantiated in the constructor or initialization method of the class.
hbaseConfig = HBaseConfiguration.create();

Incorrect example:

hbaseConfig = new Configuration();

Sharing a Configuration Instance

The HBase client code obtains the permission to interact with an HBase cluster by creating an HConnection with ZooKeeper. Each HConnection has a Configuration instance. The created HConnection instances are cached. This means that when a client needs to interact with the HBase cluster, it passes a Configuration instance to the cache. The HBase client checks whether an HConnection instance for the Configuration instance already exists in the cache. If it does not exist, a new HConnection instance is created. If it does exist, the corresponding instance is returned.

However, frequently creating new Configuration instances may lead to the creation of many unnecessary HConnection instances, which can easily reach the connection limit of ZooKeeper.

It is recommended to share a single Configuration instance throughout the client code.

Creating a Table Instance

public abstract class TableOperationImpl {
  private static Configuration conf = null;
  private static Connection connection = null;
  private static Table table = null;
  private static TableName tableName = TableName.valueOf("sample_table");

  public TableOperationImpl() {
    init();
  }
  public void init() {
    conf = ConfigurationSample.getConfiguration();
    try {
      connection = ConnectionFactory.createConnection(conf);
      table = conn.getTable(tableName);
    } catch (IOException e) {
      e.printStackTrace();
    }
  }
  public void close() {
    if (table != null) {
      try {
        table.close();
      } catch (IOException e) {
        System.out.println("Can not close table.");
      } finally {
        table = null;
      }
    }
    if (connection != null) {
      try {
        connection.close();
      } catch (IOException e) {
        System.out.println("Can not close connection.");
      } finally {
        connection = null;
      }
    }
  }
  public void operate() {
    init();
    process();
    close();
  }
}

Disallowing Concurrent Table Instance Sharing

Table is a non-thread-safe class, so the same Table instance should not be used by multiple threads simultaneously. Otherwise, concurrency issues may occur.

Caching Table Instances

Cache the Table instance that will be frequently used by a thread for a long period of time. A cached instance, however, will not be necessarily used by a thread permanently. In special circumstances, you need to rebuild a Table instance. See the next rule for details.

Correct example:

Note that the method provided here for caching Table instances using a Map is for reference only. This approach depends on the design for handling multiple threads and Table instances. If a Table instance is guaranteed to be used by only one thread and the thread has only one Table instance, there is no need to use a Map.

// This Map caches all instantiated tables with TableName as the key.
private Map<String, Table> demoTables = new HashMap<String, Table>();
// All Table instances will share this Configuration instance.
private Configuration demoConf = null;
/**
* <Initialize an HTable class.>
* <Function description>
* @param tableName
* @return
* @throws IOException
* @see [Class, Class#Method, Class#Member]
*/
private Table initNewTable(String tableName) throws IOException
{
try (Connection conn = ConnectionFactory.createConnection(demoConf)){
      return conn.getTable(tableName);
    }
}
/**
* <Get a Table instance.>
* <Function description>
* @see [Class, Class#Method, Class#Member]
*/
private Table getTable(String tableName)
{
if (demoTables.containsKey(tableName))
{
return demoTables.get(tableName);
} else {
Table table = null;
try
{
table = initNewTable(tableName);
demoTables.put(tableName, table);
}
catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
return table;
}
}
/**
* <Write data.>
* < This method does not cover optimizations for handling multiple threads and multiple Table instances in the design pattern.
* The synchronization method is used, primarily because a Table instance is not thread-safe.
* Typically, it is recommended that a Table instance be used by only one thread for writing data at a time.>
* @param dataList
* @param tableName
* @see [Class, Class#Method, Class#Member]
*/
public void putData(List<Put> dataList, String tableName)
{Table table = getTable(tableName);
// If the design ensures that no multiple threads share the same Table instance, synchronization is not required.
// Note that Table instances are not thread-safe.
synchronized (table)
{
try
{
table.put(dataList);
table.notifyAll();
}
catch (IOException e)
{
                // When an IOException is caught, the cached instance needs to be recreated.
try {
     // Close the previous connection.
       table.close();
                  // Recreate the instance.
                  table = initNewTable(tableName);
} catch (IOException e1) {
// TODO
}
}
}
}

Incorrect example:

public void putDataIncorrect(List<Put> dataList, String tableName)
{Table table = null;
try
{
// Create an HTable instance each time data is written.
table = initNewTable(tableName);
table.put(dataList);
}
catch (IOException e1)
{
// TODO Auto-generated catch block
e1.printStackTrace();
}
finally
{
table.close();
}
}

Handling Table Instance Data Write Exceptions

It is necessary to reconstruct a cashed Table instance when an IOException is detected. For details, see the example of the previous rule.

In addition, exercise caution when using the following methods:

Handling Failed Write Operations

Some data write operations may fail due to instant exceptions or process failures. Therefore, the data must be recorded so that it can be written to the HBase when the cluster is restored.

Note that the HBase client does not automatically retry when it returns data that fails to be written. It only informs the API caller of the data that failed to be written. To prevent data loss, measures must be taken to temporarily save the data in a file or in memory.

Correct example:

private List<Row> errorList = new ArrayList<Row>();
/**
* <Insert data using PutList.>
* <Synchronization is not required if this method is not called by multiple threads.>
* @param put <A single data record>
* @throws IOException
* @see [Class, Class#Method, Class#Member]
*/
public synchronized void putData(Put put)
{
// Temporarily cache the data in this list.
dataList.add(put);
// When the size of dataList reaches PUT_LIST_SIZE, perform a Put operation.
if (dataList.size() >= PUT_LIST_SIZE)
{
try
{
demoTable.put(dataList);
}
catch (IOException e)
{
// If the exception is of the RetriesExhaustedWithDetailsException type,
// it indicates that some of the data failed to be written.
// This is usually due to HBase cluster process anomalies or a large number of regions being moved,
// leading to failures after a certain number of retries.
if (e instanceof RetriesExhaustedWithDetailsException)
{
RetriesExhaustedWithDetailsException ree = 
  (RetriesExhaustedWithDetailsException)e;
int failures = ree.getNumExceptions();
for (int i = 0; i < failures; i++)
{
errorList.add(ree.getRow(i));
}
}
}
dataList.clear();
}
}

Releasing Resources

Call the Close method to release resources when the ResultScanner and Table instances are not required. To enable the Close method to be called, add the Close method to the finally block.

Correct example:

ResultScanner scanner = null;
try
{
scanner = demoTable.getScanner(s);
//Do Something here.
}
finally
{
scanner.close();
}

Incorrect example:

1. The scanner.close() method is not called in the code to release resources.
2. The scanner.close() method is not placed in the finally block.

   ResultScanner scanner = null;
   scanner = demoTable.getScanner(s);
   //Do Something here.
   scanner.close();

Adding Fault Tolerance Mechanism During Scan

Exceptions, such as lease expiration, may occur when Scan is performed. In this case, retries are recommended.

Retry is a widely applicable practice in fault tolerance of various exceptions, including those arisen from HBase operation interfaces.

Stopping the Admin Instances Promptly When They Are Not Used

Admin instances should be created when needed and stopped promptly are use. Long-term caching of the same Admin instance should be avoided.

RowKey Design Guidelines

As a distributed columnar storage engine, HBase's read/write performance and scalability heavily depend on the quality of RowKey design and the table pre-splitting strategy. Poor RowKey design can lead to data write hotspots, causing some RegionServers to be overloaded while others remain idle. Unsplit tables start with a single Region, concentrating all write requests on a single node, leading to performance bottlenecks and Region Split storms.

RowKey design principles

• Length: It is recommended that the RowKey length be 10 to 100 bytes, with a maximum of 200 bytes.

  Longer RowKeys can lead to MemStore and HFile index bloat, affecting memory utilization and query performance. The underlying HBase stores RowKeys in the index blocks of HFiles, and each row's RowKey is fully recorded. Longer RowKeys result in higher memory and disk space usage of indexes.

• Hashing: When actual data is mapped to RowKeys, it should be evenly distributed across all regions.

  This prevents read and write operations from concentrating on individual RegionServers and avoids hotspot bottlenecks. Even distribution is the prerequisite for horizontal scaling.

• Uniqueness: RowKeys must be globally unique.

  Duplicate RowKeys can lead to data overwrites and unexpected query results.

How to avoid hotspots in RowKey design

• Salting

  Add a random number or modulus to the RowKey prefix to distribute data evenly.

  Design approach:

  Raw:

  RowKey:  user123_event20260711

  Salted:

  RowKey:  0_user123_event20260711   (prefix = hash(key) % regionNum)
           1_user123_event20260711
           ...
           N_user123_event20260711

  Application scenarios:

  • Write-intensive scenarios where the write pressure needs to be evenly distributed to all regions.
  • Scenarios where range scan based on the original RowKey prefix is not required.

  Advantages and disadvantages:

  • Advantage: Write operations are evenly distributed, eliminating write hotspots.
  • Disadvantage: Scan requires traversing all prefixes, making range queries less efficient. Single data retrieval requires knowing the salted prefix value.

  Number of salted prefixes:

  • The number of prefixes must be the same as or an integer multiple of the number of pre-splits.
  • Prefix calculation formula: Prefix = Hash (service key) % N, where N is the number of prefixes.
  • Prefixes should be zero-padded fixed-length numbers, such as 00, 01, ..., and 15, to ensure correct sorting.
• Hashing

  Apply MD5 or SHA1 hashing to key fields and use the first several digits as the RowKey prefix.

  Design approach:

  Raw:

  RowKey:  13800138000

  Hashed:

  RowKey:  a1b2c_13800138000   (MD5("13800138000")[0:5])

  Application scenarios:

  • Exact queries need to be performed based on the original key, and the hash results of the same original value must be consistent.
  • Range scan based on the original key sorting is not required.

  Advantages and disadvantages:

  • Advantages: The same original value produces the same hash result, enabling the exact queries. The hash distribution is even. In addition, hashing ensures even distribution of data.
  • Disadvantages: The original sorting is lost, and range scan needs to be designed separately.

  Hashed prefix length:

  Table 1 Hashed prefix length

  Prefix Length	Value Range	Recommended Region Count
  2-digit hexadecimal	256	≤ 64
  4-digit hexadecimal	65,536	64 ~ 1,000
  6-digit hexadecimal	16,777,216	1,000 ~ 10,000
  8-digit hexadecimal	4,294,967,296	> 10,000

  The prefix should not be too long. Generally, 4 to 6 digits are sufficient for most scenarios. A longer prefix increases the hash granularity but also increases the RowKey length, potentially causing MemStore and HFile index bloat, which affects memory utilization and query performance.

• Reversing

  Store the RowKey in reverse order, moving the variable part to the front.

  Design approach:

  Raw:

  RowKey: 20260711150000_event001 (time-based prefix, which causes write hotspot)

  Reversed:

  RowKey:  100_tneve_00005111071602

  Application scenarios:

  • The RowKey prefix is a timestamp or an auto-increment sequence, and the service identifier at the end changes significantly.
  • Most queries are performed based on a single key, with fewer range scan requirements.

  Advantages and disadvantages:

  • Advantage: This approach effectively disperses hotspots caused by time or sequence prefixes. Its implementation is simple and requires no additional computation.
  • Disadvantage: Time order is lost, making range scans difficult. The RowKeys become less readable.
  • Mitigation: Reverse only the fixed-length prefix (for example, reverse the phone number) and retain the suffix sorting capability.
• (Bucket Sorting)

  Strike a balance between the hashing granularity and sorting granularity. Hashing is performed on buckets, and service keys are sorted within each bucket. Range scans are efficiently performed within each bucket, and results from different buckets are merged through parallel scans.

  Design approach:

  RowKey = <Bucket ID > + <SEP> + <Range scan key> + <SEP> + <Service key>
  
  Bucket ID = Hash (Bucketing dimension) % Number of buckets
  Range scan key = Field (such as timestamp) that requires range scanning, retaining the original order

  Example: Query behavior logs by user in a specified time range.

  RowKey = <userId Bucket ID> + <SEP> + <Timestamp> + <SEP> + <Event ID>;
  
  Example: 03_1720695600000_event001
        └┘└─────┘ └──┘
         <Bucket ID> <Timestamp in ascending order> <Event ID>
  
  Bucket ID = First two digits of MD5 (userId) in hexadecimal format → % Number of buckets (for example, 16)

  Design description:

  • Bucket ID: Hash the fixed dimension (such as user ID) of range scans, instead of the range scan key (such as timestamp).
  • Range scan key in original order: In the same bucket, data is sorted by timestamp, supporting efficient range scans.
  • Bucket count: The number of buckets should be equal to the number of regions or an integer multiple of it, ensuring even distribution of writes.

  Query method:

  Table 2 Query method for bucket-based sorting

  Query Needs	Scan Method	Scanned Regions
  Querying the behavior of a user within a specific time range	scan startRow=<Bucket number>_<startTs>, stopRow=<Bucket number>_<endTs>	1
  Querying the behavior of all users within a specific time range	Scan all buckets concurrently, with startRow/stopRow set to the time range for each bucket.	Number of buckets (parallel possible)
  Querying a specific event	get <Bucket ID>_<ts>_<Event ID>	1

  Advantages and disadvantages:

  • Advantage: Intra-bucket range scan is efficient, requiring scanning of only one region. Exact queries can be used to calculate the bucket number for locating.
  • Disadvantage: Cross-bucket range scan requires parallel scanning of multiple regions and then merging the results. When there are too many buckets, the cross-bucket scan overhead is high.
  • Constraints: The method is applicable to the scenario where there is a fixed grouping dimension (such as user ID, device ID, and tenant ID) during the range scanning.
  1. Bucket pre-splitting design based on service keys

     Application scenarios: The range scan key is the service primary key (such as the order No. prefix and device type code), and no additional hashing is required.

     Design approach:

     RowKey = <Service key prefix> + <SEP> + <Service key> + <SEP> + <Timestamp>
     
     Service key prefix = First N digits of the service key (for example, the first four digits of the order No. and device type code)

     Example: Bucketing by order No. prefix

     RowKey = <First four digits of the order No.> + <SEP> + <Order No.> + <SEP> + <Timestamp>
     
     Example: ORD2_ORD202607110001_1720695600000
            └┘ └──────┘└─────┘
            <Prefix bucket> <Order No.> <Timestamp>
     
     Pre-splitting based on the order No. prefix range:
       SPLITS => ['ORD1', 'ORD2', 'ORD3', 'ORD4', 'ORD5']

     Design description:

     • Bucketing by service key prefix: Orders with the same prefix naturally fall into the same region and can be scanned by prefix range.
     • Pre-splitting by service key prefix: The split boundaries are aligned with service key prefixes to avoid data skew.
     • Constraints: The values of service key prefixes are evenly distributed, and the prefixes have business meanings. (The prefixes can be queried by prefix range.)

     Note:

     • If the distribution of service key prefixes is uneven (for example, some prefixes have significantly more orders), additional hashing can be applied to redistribute the prefixes uniformly.
     • You can use hash (prefix) % N to redistribute uneven prefixes into evenly distributed buckets.
  2. Time-series range scan design

     Application scenarios: The queries are mainly based on the time range and data needs to be scanned by time segment.

     Core challenges: Time prefixes are naturally ordered but can cause write hotspots (data of the current time is written to the last region in a batch).

     Solution 1: Time-based bucketing + Intra-bucket time ordering (recommended)

     RowKey = <Time bucket> + <SEP> + <Exact timestamp> + <SEP> + <Service ID>
     
     Time bucket = Timestamp rounded up by hour/day
     Example (bucketing by day): 20260711_1720695600000_device001
                        └────┘└───┘ └───┘
                        <Date bucket>  <Exact timestamp>  <Device ID>

     • Data of the same day is stored in the same bucket (region) and sorted by time, supporting efficient time range scans.
     • During data writing, only one region receives data writes for the current day, but daily hotspotting is acceptable (the write pressure is distributed by day).
     • Application scenarios: Write QPS < 50,000/day, or daily archiving and query.

     Solution 2: Time prefix + Hashed sub-bucket

     Time prefix + Hashed sub-bucket**

     RowKey = <Time bucket > + <SEP> + <Hashed sub-bucket> + <SEP> + <Exact timestamp> + <SEP> + <Service ID>
     
     Example: 20260711_a3_1720695600000_device001
            └──┘└┘└─────┘└───┘
            <Date bucket> <Sub-bucket> <Exact timestamp> <Device ID>
     
     Hashed sub-bucket = Hash (Service ID) % Number of sub-buckets (for example, 8)

     • Data of the same day is further distributed to multiple regions based on hashed sub-buckets.
     • During time range scan, all sub-buckets of the same day are scanned concurrently.
     • Application scenarios: Write QPS > 50,000/day or scenarios with high write volume per day.
• Rowkey solution selection procedure
  1. Check whether the service involves sequential range scan.
     • If yes, go to 3.
     • If no, go to 2.
  2. Check whether the write frequency is high (greater than 100,000 QPS).
     • If yes, refer to Salting (Number of prefixes = Number of regions).
     • If no, refer to the Hashing prefix (mainly for exact queries).
  3. Check the service data scan dimension.
     • If the time range is used, go to 4.
     • If the service key range is used, go to 5.
     • If the fixed dimension and range dimension are used together, see Bucket Sorting.
  4. Check the write rate.
     • If the write QPS is less than 50,000/day, use time-based bucketing + intra-bucket time ordering.
     • If the write QPS is greater than or equal to 50,000/day, use time prefix + hashed sub-buckets.
  5. Check the distribution of service key prefixes.
     • If service key prefixes are evenly distributed, use the bucket pre-splitting design based on service keys.
     • If service key prefixes are unevenly distributed, use hash (prefix) bucketing + intra-bucket sorting.