Help Center> MapReduce Service> Component Operation Guide (LTS)> Using HBase> Common Issues About HBase> What Can I Do When HBase Fails to Recover a Task and a Message Is Displayed Stating "Rollback recovery failed"?
Updated on 2023-04-28 GMT+08:00

What Can I Do When HBase Fails to Recover a Task and a Message Is Displayed Stating "Rollback recovery failed"?

Question

The system automatically rolls back data after an HBase recovery task fails. If "Rollback recovery failed" is displayed, the rollback fails. After the rollback fails, data stops being processed and the junk data may be generated. How can I resolve this problem?

Answer

You need to manually clear the junk data before performing the backup or recovery task next time.

  1. Install the cluster client in /opt/client.
  2. Run source /opt/client/bigdata_env as the client installation user to configure environment variables.
  3. Run the kinit admin command.
  4. Run zkCli.sh -server business IP address of ZooKeeper:2181 to connect to the ZooKeeper.
  5. Run deleteall /recovering to delete the junk data. Run quit to disconnect ZooKeeper.

    Running this command will cause data loss. Exercise caution.

  6. Run hdfs dfs -rm -f -r /user/hbase/backup to delete temporary data.
  7. Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Backup and Restoration > Restoration Management. In the task list, locate the row that contains the target task and click View History in the Operation column. In the displayed dialog box, click before a specified execution record to view the snapshot name.

    Snapshot [ snapshot name ] is created successfully before recovery.

  8. Switch to the client, run hbase shell, and then delete_all_snapshot 'snapshot name.*' to delete the temporary snapshot.