Backing Up Flink Metadata

Scenarios

To ensure Flink metadata security or before a major operation on Flink (such as upgrade or migration), you need to back up Flink metadata. The backup data can be used to recover the system if an exception occurs or the operation has not achieved the expected result, minimizing the adverse impacts on services.

Flink metadata backup tasks can be created on FusionInsight Manager. The system supports both automatic and manual backup modes.

MRS clusters support multiple data path types for backing up Flink metadata.

• LocalDir: indicates that backup files are stored on the local disk of the active management node, with the standby management node automatically synchronizing the backup files.

  The default storage directory is Data storage path/LocalBackup/, for example, /srv/BigData/LocalBackup.

• LocalHDFS: indicates that backup files are stored in the HDFS directory of the current cluster.
• RemoteHDFS: indicates that backup files are stored in the HDFS directory of the standby cluster.
• OBS: indicates that backup files are stored in OBS.

Notes and Constraints

MRS 3.5.0 and later versions support the backup of Flink metadata to OBS.

Prerequisites

• To back up data to a remote HDFS, the following conditions must be met:
  • A standby cluster for backing up data has been created. Its authentication mode must be the same as that of the active cluster.
  • If the active cluster is deployed in security mode and the active and standby clusters are not managed by the same FusionInsight Manager, mutual trust has been configured. For details, see Configuring Mutual Trust Between MRS Clusters. If the active cluster is deployed in normal mode, mutual trust is not required.
  • Cross-cluster replication has been configured for the active and standby clusters. For details, see Enabling MRS Inter-Cluster Replication.
  • Time is consistent between the active and standby clusters, with the NTP services on both clusters configured to use the same time source.
• The HDFS and Yarn services have been installed if data needs to be backed up to HDFS.
• The backup type, period, policy, and other specifications have been planned based on the service requirements and you have checked whether Data storage path/LocalBackup/ has sufficient space on the active and standby management nodes.
• If you want to back up data to OBS, you have connected the current cluster to OBS and have the permission to access OBS. Only MRS 3.5.0 or later supports data backup to OBS.

Backing Up Flink Metadata

1. Log in to MRS Manager.

   For details about how to log in to MRS Manager, see Accessing MRS Manager.

2. Choose O&M > Backup and Restoration > Backup Management.
3. Click Create.
4. Set Name to the name of the backup task.
5. Select the cluster to be operated from Backup Object.
6. Set Mode to the type of the backup task. Periodic indicates that the backup task is periodically executed. Manual indicates that the backup task is manually executed.

   Table 1 Periodic backup parameters

   Parameter	Example Value	Description
   Started	2024/11/22 10:29:11	Time when the task should start for the first time.
   Period	1 day	Task execution interval. Supported values are Hours and Days.
   Backup Policy	Full backup every time	Full backup at the first time and incremental backup subsequently Full backup every time Full backup once every n times Incremental backup is not supported for Manager data and component metadata. Only Full backup every time is supported. If Path Type is set to NFS or CIFS, incremental backup cannot be used. When incremental backup is used for NFS or CIFS backup, the latest full backup data is updated each time the incremental backup is performed. Therefore, no new recovery point is generated.

7. In Configuration, select Flink under Metadata and other data.
8. Select a backup directory type for Path Type of Flink.

   Table 2 Path of backup data

   Path Type	Parameter	Description
   LocalDir	Maximum Number of Backup Copies	Number of backup files that can be retained in the backup directory.
   LocalHDFS	Target Path	HDFS directory for storing the backup files. The path cannot be configured as an HDFS hidden directory, such as a snapshot or recycle bin directory. The path cannot be configured as a default system directory, such as /hbase or /user/hbase/backup.
   	Maximum Number of Backup Copies	Number of backup files that can be retained in the backup directory.
   	Target NameService Name	NameService name of the backup directory. The default value is hacluster.
   RemoteHDFS	Destination NameService Name	Name of the NameService in the standby cluster. You can set it to the NameService name (haclusterX, haclusterX1, haclusterX2, haclusterX3, or haclusterX4) of the built-in remote cluster. You can also set it to the NameService name of a configured remote cluster.
   	IP Mode	IP version of the target IP address. The system automatically determines the IP version, such as IPv4 or IPv6, based on the cluster network type.
   	Target NameNode IP Address	Service plane IP address of the active or standby NameNode in the standby cluster.
   	Target Path	HDFS directory in the standby cluster for storing backup data. The path cannot be configured as an HDFS hidden directory, such as a snapshot or recycle bin directory. The path cannot be configured as a default system directory, such as /hbase or /user/hbase/backup.
   	Maximum Number of Backup Copies	Number of backup files that can be retained in the backup directory.
   	Queue Name	Name of the YARN queue used for backup task execution. The name must be identical to that of a queue currently running properly in the cluster.
   OBS (available in MRS 3.5.0 and later)	Target Path	OBS directory for storing backup data.
   	Maximum Number of Backup Copies	Number of backup files that can be retained in the backup directory.

9. Click OK.
10. In the Operation column of the created task in the backup task list, click More and select Back Up Now to execute the backup task.

    After the backup task is executed, the system automatically creates a subdirectory for each backup task in the backup directory. The format of the subdirectory name is Backup task name_Task creation time, and the subdirectory is used to save data source backup files. The format of the backup file name is Data source_Task execution time.tar.gz.

Helpful Links

Backup Flink metadata may be restored if data is accidentally modified, deleted, or requires retrieval; if major Flink operations (such as upgrades or migration) cause exceptions in system data or fail to achieve the expected result; if all Flink modules fail and become unavailable; or if data needs to be migrated to a new cluster. For detailed operations, refer to Restoring Flink Metadata.