HBase Fails to Recover a Task

Symptom

The system automatically rolls back data after an HBase recovery task fails. If "Rollback recovery failed" is displayed, the rollback fails. After the rollback fails, data stops being processed and the junk data may be generated. How can I resolve this problem?

Procedure

You need to manually clear the junk data before performing the backup or recovery task next time.

1. Install the cluster client in /opt/client.
2. Run the following command as the client installation user to configure environment variables.

   source /opt/client/bigdata_env

3. Run the following command to authenticate the user:

   kinit admin

4. Run the following command to connect to the ZooKeeper.

   zkCli.sh -server business IP address of ZooKeeper:2181

5. Run the following command to delete the junk data.

   deleteall /recovering

   Running this command will cause data loss. Exercise caution.

   Run quit to disconnect ZooKeeper.

6. Run the following command to delete temporary data.

   hdfs dfs -rm -f -r /user/hbase/backup

7. Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Backup and Restoration > Restoration Management. In the task list, locate the row that contains the target task and click View History in the Operation column. In the displayed dialog box, click before a specified execution record to view the snapshot name.

   Snapshot [ snapshot name ] is created successfully before recovery.

8. Switch to the client, run hbase shell, and then run the following command to delete the temporary snapshot.

   delete_all_snapshot 'snapshot name.*'