ALM-19036 Bad Blocks Exist in HBase Key Directory Data

Alarm Description

The system checks for bad blocks in HBase key directories every 5 minutes, including the hbase.version file and the hbase:meta and master:store table directories. This alarm is generated when a bad block is detected.

This alarm is cleared when the system detects that no bad blocks exist in the HBase key directories.

This alarm applies only to MRS 3.5.0 or later.

Alarm Attributes

Alarm Parameters

Impact on the System

If block loss occurs in HBase key directories, the HBase service becomes unavailable, causing service request stacking or interruptions.

Possible Causes

HDFS is faulty.

Handling Procedure

Handling bad blocks in key directory data involves operations that quickly restore the HBase service, such as stopping the HBase service. Be mindful that these operations will interrupt services. Also, be aware of any data stacking on the services.

Check whether the HDFS service is normal.

1. Log in to FusionInsight Manager, choose Cluster > Services > HDFS, and check whether the HDFS running status is Normal.
   • If yes, go to 3.
   • If no, go to 2.

2. Restore the HDFS running status to Normal by following the instructions provided in the alarm help, and go to 3.

Rebuild key directory data.

3. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, check the cause of the alarm whose Alarm ID is 19036, and identify the directory with block loss.
4. Perform the restoration operations based on the directory where block loss has occurred.
   • If Alarm Cause indicates that bad blocks exist in the hbase.version file, go to 5 to restore the hbase.version file.
   • If Alarm Cause indicates that bad blocks exist in the hbase:meta table directory, go to 6 to restore the hbase meta table directory files.
   • If Alarm Cause indicates that bad blocks exist in the master:store table directory, go to 7 to restore the master store table directory files.

5. Restore the hbase.version file.
   1. Log in to the node where the client is installed as the client installation user.
   2. Run the following command to go to the client installation directory:

      cd Client installation directory

   3. Run the following commands to configure environment variables:

      source bigdata_env

      source HBase/component_env

   4. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.

      kinit hbase

      Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:

      export HADOOP_USER_NAME=hbase

   5. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:

      hdfs dfs -mkdir /tmp/hbase_bak

   6. Run the following command to back up the old file:

      hdfs dfs -mv /hbase/hbase.version /tmp/hbase_bak

   7. Run the following command to restore the hbase.version file:

      hbase hbck -j ${HBASE_HOME}/tools/hbase-hbck2-*.jar filesystem -fixVersionFile

      • After the command is executed successfully, run the following command to view the restored hbase.version file:

        hdfs dfs -ls /hbase

      • If the command fails to be executed, go to 8.
   8. Log in to FusionInsight Manager, choose Cluster > Services > HBase > Instances, select all HMaster instances, click More, and select Instance Rolling Restart. In the dialog box that is displayed, enter the password of the current user, and click OK to perform a rolling restart of all HMaster instances.
   9. After the HMaster instances are restarted, check whether the alarm is cleared in the alarm list.
      • If yes, no further action is required.
      • If no, go to 8.

6. Restore the hbase meta table directory files.
   1. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Stop Service in the upper right corner of the Dashboard page. In the dialog box that is displayed, enter the password of the current user and click OK to stop the HBase service.
   2. Log in to the node where the client is installed as the client installation user.
   3. Run the following command to go to the client installation directory:

      cd Client installation directory

   4. Run the following commands to configure environment variables:

      source bigdata_env

      source HBase/component_env

   5. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.

      kinit hbase

      Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:

      export HADOOP_USER_NAME=hbase

   6. Run the following commands to regenerate the meta table data:

      export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${HBASE_HOME}/tools/*

      hbase org.apache.hbase.hbck1.OfflineMetaRepair -details

      If Success is displayed, the command is successfully executed. In this case, go to 6.g. If the command fails to be executed, go to 8.

   7. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:

      hdfs dfs -mkdir /tmp/hbase_bak

   8. Run the following command to back up and clear HMaster data:

      hdfs dfs -mv /hbase/MasterData/* /tmp/hbase_bak

   9. Run the following commands to clear the location information of the meta table:

      hbase zkcli

      deleteall /hbase/meta-region-server

      quit

   10. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Start Service in the upper right corner of the Dashboard page to start the HBase service.
   11. After the HBase service is started, check whether the alarm is cleared in the alarm list.
       • If yes, no further action is required.
       • If no, go to 8.

7. Restore the master store table directory files.
   1. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Stop Service in the upper right corner of the Dashboard page. In the dialog box that is displayed, enter the password of the current user and click OK to stop the HBase service.
   2. Log in to the node where the client is installed as the client installation user.
   3. Run the following command to go to the client installation directory:

      cd Client installation directory

   4. Run the following commands to configure environment variables:

      source bigdata_env

      source HBase/component_env

   5. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.

      kinit hbase

      Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:

      export HADOOP_USER_NAME=hbase

   6. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:

      hdfs dfs -mkdir /tmp/hbase_bak

   7. Run the following command to back up and clear HMaster data:

      hdfs dfs -mv /hbase/MasterData/* /tmp/hbase_bak

   8. Run the following commands to clear the location information of the meta table:

      hbase zkcli

      deleteall /hbase/meta-region-server

      quit

   9. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Start Service in the upper right corner of the Dashboard page to start the HBase service.
   10. After the HBase service is started, check whether the alarm is cleared in the alarm list.
       • If yes, no further action is required.
       • If no, go to 8.

After the HBase service is restored, observe the service for a period of time. After confirming that HBase and related services are normal, you are advised to run the following command to delete the backup directory to prevent residual useless files:

hdfs dfs -rm -r /tmp/hbase_bak

Collect fault information.

8. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
9. Expand the Service drop-down list, and select HBase for the target cluster.
10. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
11. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.