Restoring Manager Data (MRS 3.x and Later Versions)

Scenario

Manager data needs to be recovered in the following scenarios: data is modified or deleted unexpectedly and needs to be restored. After an administrator performs critical data adjustment in FusionInsight Manager, an exception occurs or the operation has not achieved the expected result. All modules are faulty and become unavailable.

System administrators can create a restoration task in FusionInsight Manager to recover Manager data. Only manual restoration tasks are supported.

• Data restoration can be performed only when the system version is consistent with that of data backup.
• To recover data when the service is running properly, you are advised to manually back up the latest management data before recovering data. Otherwise, the Manager data that is generated after the data backup and before the data restoration will be lost.

Impact on the System

• In the restoration process, the Controller needs to be restarted and FusionInsight Manager cannot be logged in or operated during the restart.
• In the restoration process, all clusters need to be restarted and cannot be accessed during the restart.
• After data restoration, the data, such as system configuration, user information, alarm information, and audit information, that is generated after the data backup and before the data restoration will be lost. This may result in data query failure or cluster access failure.
• After the Manager data is recovered, the system forces the LdapServer of each cluster to synchronize data from the OLadp.

Prerequisites

• To restore data from a remote HDFS, the following conditions must be met:
  • Prepare a standby cluster for restoring data, and ensure that data in this cluster has been backed up. For details, see Backing Up Manager Data (MRS 3.x and Later Versions). If the active cluster is deployed in security mode and the active and standby clusters are not managed by the same FusionInsight Manager, system mutual trust needs to be configured. For details, see Configuring Mutual Trust Between MRS Clusters. If the active cluster is deployed in normal mode, no mutual trust is required.
  • Cross-cluster replication has been configured for the active and standby clusters. For details, see Enabling MRS Inter-Cluster Replication.
  • Time is consistent between the active and standby clusters and the NTP services on the active and standby clusters use the same time source.

• The status of the OMS resources and the LdapServer instances of each cluster is normal. If the status is abnormal, data restoration cannot be performed.
• The status of the cluster hosts and services is normal. If the status is abnormal, data restoration cannot be performed.
• The cluster host topologies during data restoration and data backup are the same. If the topologies are different, data restoration cannot be performed and you need to back up data again.
• The services added to the cluster during data restoration and data backup are the same. If the topologies are different, data restoration cannot be performed and you need to back up data again.
• The upper-layer applications that depend on the cluster are stopped.

Restoring Manager Data

1. On FusionInsight Manager, choose O&M > Backup and Restoration > Backup Management.
2. In the Operation column of a specified task in the task list, choose More > View History to view the historical backup task execution records.

   In the displayed window, locate a specified success record and click View in the Backup Path column to view the backup path information of the task and find the following information:

   • Backup Object specifies the data source of the backup data.
   • Backup Path specifies the full path where the backup files are saved.

     Select the correct item, and manually copy the full path of backup files in Backup Path.

3. On FusionInsight Manager, choose O&M > Backup and Restore > Restoring Management. On the displayed page, click Create.
   Figure 1 Creating a restoration task
   

4. Set Task Name to the name of the restoration task.
5. Set Recovery Object to OMS.
6. Select OMS.
7. Set Path Type of OMS to a backup directory type.

   The settings vary according to backup directory types:

   • LocalDir: indicates that the backup files are stored on the local disk of the active management node.

     If you select LocalDir, you also need to set Source Path to select the backup file to be restored, for example, Version_Data source_Task execution time.tar.gz.

   • LocalHDFS: indicates that the backup files are stored in the HDFS directory of the current cluster.
     If you select LocalHDFS, set the following parameters:

     • Source Path: indicates the full path of the backup file in the HDFS, for example, Backup path/Backup task name_Task creation time/Version_Data source_Task execution time.tar.gz.
     • Cluster for Restoration: Enter the name of the cluster used during restoration task execution.
     • Source NameService Name: indicates the NameService name that corresponds to the backup directory when a restoration task is executed. The default value is hacluster.
   • RemoteHDFS: indicates that the backup files are stored in the HDFS directory of the standby cluster.
     If you select RemoteHDFS, set the following parameters:

     • Source NameService Name: indicates the NameService name of the backup data cluster. You can enter the built-in NameService name of the remote cluster, for example, haclusterX, haclusterX1, haclusterX2, haclusterX3, or haclusterX4. You can also enter a configured NameService name of the remote cluster.
     • IP Mode: indicates the mode of the target IP address. The system automatically selects the IP address mode based on the cluster network type, for example, IPv4 or IPv6.
     • Source NameNode IP Address: indicates the NameNode service plane IP address of the standby cluster, supporting the active node or standby node.
     • Source Path: indicates the full path of HDFS directory for storing backup data of the standby cluster, for example, Backup path/Backup task name_Data source_Task creation time/Version_Data source_Task execution time.tar.gz.
     • Source Cluster: Select the cluster of the Yarn queue used by the recovery data.
     • Queue Name: indicates the name of the Yarn queue used for backup task execution. The name must be the same as the name of the queue that is running properly in the cluster.
   • NFS: indicates that backup files are stored in the NAS using the NFS protocol. If you select NFS, set the following parameters:
     • IP Mode: indicates the mode of the target IP address. The system automatically selects the IP address mode based on the cluster network type, for example, IPv4 or IPv6.
     • Server IP Address: indicates the IP address of the NAS server.
     • Source Path: indicates the complete path of the backup file on the NAS server, for example, Backup path/Backup task name_Data source_Task creation time/Version_Data source_Task execution time.tar.gz.
   • CIFS: indicates that backup files are stored in the NAS using the CIFS protocol. If you select CIFS, set the following parameters:
     • IP Mode: indicates the mode of the target IP address. The system automatically selects the IP address mode based on the cluster network type, for example, IPv4 or IPv6.
     • Server IP Address: indicates the IP address of the NAS server.
     • Port: indicates the port number used to connect to the NAS server over the CIFS protocol. The default value is 445.
     • Username: indicates the username set when the CIFS protocol is configured.
     • Password: indicates the password set when the CIFS protocol is configured.
     • Source Path: indicates the full path of the backup file on the NAS server, for example, Backup path/Backup task name_Data source_Task creation time/Version_Data source_Task execution time.tar.gz.
   • SFTP: indicates that backup files are stored in the server using the SFTP protocol.
     If you select SFTP, set the following parameters:

     • IP Mode: indicates the mode of the target IP address. The system automatically selects the IP address mode based on the cluster network type, for example, IPv4 or IPv6.

     • Server IP Address: indicates the IP address of the server where the backup data is stored.
     • Port: indicates the port number used to connect to the backup server over the SFTP protocol. The default value is 22.
     • Username: indicates the username for connecting to the server using the SFTP protocol.
     • Password: indicates the password for connecting to the server using the SFTP protocol.
     • Source Path: indicates the full path of the backup file on the backup server, for example, Backup path/Backup task name_Data source_Task creation time/Version_Data source_Task execution time.tar.gz.
   • OBS: indicates that backup files are stored in OBS.

     If you select OBS, set the following parameters:

     • Source Path: indicates the full OBS path of a backup file, for example, Backup path/Backup task name_Data source_Task creation time/Version_Data source_Task execution time.tar.gz.
       

       Only MRS 3.1.0 or later supports saving backup files in OBS.

8. Click OK.
9. In the restoration task list, locate a created task and click Start in the Operation column to execute the restoration task.
   • After the restoration is successful, the progress bar is in green.
   • After the restoration is successful, the restoration task cannot be executed again.
   • If the restoration task fails during the first execution, rectify the fault and click Retry to execute the task again.

10. Log in to the active and standby management nodes as user omm.
11. Run the following command to restart OMS:

    sh ${BIGDATA_HOME}/om-server/om/sbin/restart-oms.sh

    The command is run successfully if the following information is displayed:

    start HA successfully.

    Run sh ${BIGDATA_HOME}/om-server/om/sbin/status-oms.sh to check whether HAAllResOK of the management node is Normal and whether FusionInsight Manager can be logged in again. If yes, OMS is restarted successfully.

12. On FusionInsight Manager, click Cluster, click the name of the target cluster, and choose Services > KrbServer. On the displayed page, choose More > Synchronize Configuration, click OK, and wait for the KrbServer configuration to be synchronized and the service to be restarted.
13. Choose Cluster, click the name of the desired cluster, and choose More > Synchronize Configurations, click OK, and wait until the cluster configuration is synchronized successfully.
14. On FusionInsight Manager, click Cluster, click the name of the target cluster, and choose More > Restart. On the displayed page, enter the password of the current login user, click OK, and wait for the cluster to be restarted.