On this page

Show all

Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-19036 Bad Blocks Exist in HBase Key Directory Data

ALM-19036 Bad Blocks Exist in HBase Key Directory Data

Updated on 2024-11-13 GMT+08:00

View PDF

Alarm Description

The system checks for bad blocks in HBase key directories every 5 minutes, including the hbase.version file and the hbase:meta and master:store table directories. This alarm is generated when a bad block is detected.

This alarm is cleared when the system detects that no bad blocks exist in the HBase key directories.

This alarm applies only to MRS 3.5.0 or later.

Alarm Attributes

Alarm ID	Alarm Severity	Auto Cleared
19036	Critical	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm was generated.
	ServiceName	Specifies the service for which the alarm was generated.
	RoleName	Specifies the role for which the alarm was generated.
	HostName	Specifies the host for which the alarm was generated.

Impact on the System

If block loss occurs in HBase key directories, the HBase service becomes unavailable, causing service request stacking or interruptions.

Possible Causes

HDFS is faulty.

Handling Procedure

NOTICE:

Handling bad blocks in key directory data involves operations that quickly restore the HBase service, such as stopping the HBase service. Be mindful that these operations will interrupt services. Also, be aware of any data stacking on the services.

Check whether the HDFS service is normal.

Log in to FusionInsight Manager, choose Cluster > Services > HDFS, and check whether the HDFS running status is Normal.
- If yes, go to 3.
- If no, go to 2.
Restore the HDFS running status to Normal by following the instructions provided in the alarm help, and go to 3.

Rebuild key directory data.

Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, check the cause of the alarm whose Alarm ID is 19036, and identify the directory with block loss.
Perform the restoration operations based on the directory where block loss has occurred.
- If Alarm Cause indicates that bad blocks exist in the hbase.version file, go to 5 to restore the hbase.version file.
- If Alarm Cause indicates that bad blocks exist in the hbase:meta table directory, go to 6 to restore the hbase meta table directory files.
- If Alarm Cause indicates that bad blocks exist in the master:store table directory, go to 7 to restore the master store table directory files.
Restore the hbase.version file.
1. Log in to the node where the client is installed as the client installation user.
2. Run the following command to go to the client installation directory:
  cd Client installation directory
3. Run the following commands to configure environment variables:
  source bigdata_env
  
  source HBase/component_env
4. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.
  kinit hbase
  
  Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:
  
  export HADOOP_USER_NAME=hbase
5. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:
  hdfs dfs -mkdir /tmp/hbase_bak
6. Run the following command to back up the old file:
  hdfs dfs -mv /hbase/hbase.version /tmp/hbase_bak
7. Run the following command to restore the hbase.version file:
  hbase hbck -j ${HBASE_HOME}/tools/hbase-hbck2-*.jar filesystem -fixVersionFile
  - After the command is executed successfully, run the following command to view the restored hbase.version file:
    hdfs dfs -ls /hbase
  - If the command fails to be executed, go to 8.
8. Log in to FusionInsight Manager, choose Cluster > Services > HBase > Instances, select all HMaster instances, click More, and select Instance Rolling Restart. In the dialog box that is displayed, enter the password of the current user, and click OK to perform a rolling restart of all HMaster instances.
9. After the HMaster instances are restarted, check whether the alarm is cleared in the alarm list.
  - If yes, no further action is required.
  - If no, go to 8.
Restore the hbase meta table directory files.
1. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Stop Service in the upper right corner of the Dashboard page. In the dialog box that is displayed, enter the password of the current user and click OK to stop the HBase service.
2. Log in to the node where the client is installed as the client installation user.
3. Run the following command to go to the client installation directory:
  cd Client installation directory
4. Run the following commands to configure environment variables:
  source bigdata_env
  
  source HBase/component_env
5. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.
  kinit hbase
  
  Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:
  
  export HADOOP_USER_NAME=hbase
6. Run the following commands to regenerate the meta table data:
  export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${HBASE_HOME}/tools/*
  
  hbase org.apache.hbase.hbck1.OfflineMetaRepair -details
  
  If Success is displayed, the command is successfully executed. In this case, go to 6.g. If the command fails to be executed, go to 8.
7. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:
  hdfs dfs -mkdir /tmp/hbase_bak
8. Run the following command to back up and clear HMaster data:
  hdfs dfs -mv /hbase/MasterData/* /tmp/hbase_bak
9. Run the following commands to clear the location information of the meta table:
  hbase zkcli
  
  deleteall /hbase/meta-region-server
  
  quit
10. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Start Service in the upper right corner of the Dashboard page to start the HBase service.
11. After the HBase service is started, check whether the alarm is cleared in the alarm list.
  - If yes, no further action is required.
  - If no, go to 8.
Restore the master store table directory files.
1. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Stop Service in the upper right corner of the Dashboard page. In the dialog box that is displayed, enter the password of the current user and click OK to stop the HBase service.
2. Log in to the node where the client is installed as the client installation user.
3. Run the following command to go to the client installation directory:
  cd Client installation directory
4. Run the following commands to configure environment variables:
  source bigdata_env
  
  source HBase/component_env
5. Kerberos authentication is enabled for the cluster (the cluster is in security mode): Run the following command to authenticate as the HBase built-in user. If this is your first-time authentication, enter the default password and change it.
  kinit hbase
  
  Kerberos authentication is disabled for the cluster (the cluster is in normal mode): Run the following command to set the Hadoop username:
  
  export HADOOP_USER_NAME=hbase
6. Run the following command to create a backup directory that does not exist in HDFS, for example, /tmp/hbase_bak:
  hdfs dfs -mkdir /tmp/hbase_bak
7. Run the following command to back up and clear HMaster data:
  hdfs dfs -mv /hbase/MasterData/* /tmp/hbase_bak
8. Run the following commands to clear the location information of the meta table:
  hbase zkcli
  
  deleteall /hbase/meta-region-server
  
  quit
9. Log in to FusionInsight Manager, choose Cluster > Services > HBase, and click Start Service in the upper right corner of the Dashboard page to start the HBase service.
10. After the HBase service is started, check whether the alarm is cleared in the alarm list.
  - If yes, no further action is required.
  - If no, go to 8.

NOTE:

After the HBase service is restored, observe the service for a period of time. After confirming that HBase and related services are normal, you are advised to run the following command to delete the backup directory to prevent residual useless files:

hdfs dfs -rm -r /tmp/hbase_bak

Collect fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select HBase for the target cluster.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.

Parent topic: MRS Cluster Alarm Handling Reference

Previous topic: ALM-19035 Size of the RegionServer Call Queue Exceeds the Threshold

Next topic: ALM-20002 Hue Service Unavailable

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot