Region in the RIT State for a Long Time Due to HBase File Loss

Issue

After data is migrated to a new cluster, the region status of HBase is RIT or FAILED_OPEN.

Cause Analysis

1. Log in to the core node of the cluster and check the RegionServer log. The following errors can be found:
   • Failed to open region 3b3ae24c65fc5094bc2acfebaa7a56de, and the region status is FAILED_OPEN.
   • A file in region b7b3faab86527b88a92f2a248a54d3dc cannot be found and error FileNotFoundException is displayed.
2. Run the hbase hbck command on the HBase client to check the region. The following error message is displayed:

   ERROR: Found lingering reference file hdfs://hacluster/hbase/news_user_actions/3b3ae24c65fc5094bc2acfebaa7a56de/meta/0f47cda55fa44cf9aa2599079894aed6.b7b3faab86527b88a92f2a248a54d3dc

   According to the reference file, region 3b3ae24c65fc5094bc2acfebaa7a56de is a child region of region b7b3faab86527b88a92f2a248a54d3dc and has referenced the file of its parent region. However, the file of the parent region is lost during the migration, and the child region cannot find the reference file.

   This exception cannot be resolved by retries. Therefore, the region is in the RIT state for a long period.

Procedure

1. On the HDFS client, run the following command to delete the residual file queried in 2:

   hdfs dfs -rmr HDFS path of the residual file

   Example:

   hdfs dfs -rmr /hbase/news_user_actions/3b3ae24c65fc5094bc2acfebaa7a56de/meta/0f47cda55fa44cf9aa2599079894aed6.b7b3faab86527b88a92f2a248a54d3dc