Configuring HDFS Mover to Migrate Data

Scenario

Mover is a data migration tool whose working mode is similar to that of the HDFS Balancer. Mover can redistribute data in the cluster based on the configured data storage policy.

Use Mover to periodically check whether the specified HDFS file or directory in the HDFS file system meets the preset storage policy. If not, migrate data to make them meet the policy.

Notes and Constraints

This section applies to MRS 3.x or later.

Configuring Automatic Data Migration

1. Log in to FusionInsight Manager.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Choose Cluster > Services > HDFS > Configurations > All Configurations.
3. Search for the following parameters and change their values as required.

   Table 1 Parameters

   Parameter	Description	Default Value
   dfs.mover.auto.enable	Determines whether to enable the data replica migration function. true: The function is enabled. false: The function is disabled.	false
   dfs.mover.auto.cron.expression	Specifies the CRON expression for HDFS automatic data migration, and is used to control the start time of data migration. This parameter is valid only when dfs.mover.auto.enable is set to true. The default value is 0 * * * *, indicating that the task is executed at the top of each hour. For details about the expression, see Table 2.	0 * * * *
   dfs.mover.auto.hdfsfiles_or_dirs	Specifies HDFS file and directory lists that implement automatic replica migration in specified clusters. Separate the lists with spaces. This parameter is valid only when dfs.mover.auto.enable is set to true.	-

   Table 2 CRON expressions

   Column	Description
   1	Minute. The value ranges from 0 to 59.
   2	Hour. The value ranges from 0 to 23.
   3	Date. The value ranges from 1 to 31.
   4	Month. The value ranges from 1 to 12.
   5	Week. The value ranges from 0 to 6. 0 indicates Sunday.

4. Click Save to make configurations take effect. You do not need to restart the HDFS service.

Manually Migrating Data

1. Install the client. If the client has been installed, skip this step.

   For example, the installation directory is /opt/client. You need to change it to the actual installation directory.

   For details about how to download and install the cluster client, see Installing an MRS Cluster Client.

2. Log in to the node where the client is installed as the client installation user.
3. Go to the client installation directory, for example, /opt/client.

   cd /opt/client

4. Run the following command to configure environment variables:

   source bigdata_env

5. If the cluster is in security mode, run the following command to authenticate the user. If the cluster is in normal mode, skip this step. The user must have the supergroup permission.

   kinit Component service user

6. Run the mover command in the following format:

   hdfs mover -p <HDFS file or directory>

   -p: specifies the HDFS file or directory from which data is to be migrated. You can specify a single file or a directory. If a directory is specified, all files in the directory will be migrated.