Configuring HDFS Mover to Migrate Data
Scenario
Mover is a data migration tool whose working mode is similar to that of the HDFS Balancer. Mover can redistribute data in the cluster based on the configured data storage policy.
Use Mover to periodically check whether the specified HDFS file or directory in the HDFS file system meets the preset storage policy. If not, migrate data to make them meet the policy.
Notes and Constraints
This section applies to MRS 3.x or later.
Configuring Automatic Data Migration
- Log in to FusionInsight Manager.
For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.
- Choose Cluster > Services > HDFS > Configurations > All Configurations.
- Search for the following parameters and change their values as required.
Table 1 Parameters Parameter
Description
Default Value
dfs.mover.auto.enable
Determines whether to enable the data replica migration function.
- true: The function is enabled.
- false: The function is disabled.
false
dfs.mover.auto.cron.expression
Specifies the CRON expression for HDFS automatic data migration, and is used to control the start time of data migration.
This parameter is valid only when dfs.mover.auto.enable is set to true. The default value is 0 * * * *, indicating that the task is executed at the top of each hour. For details about the expression, see Table 2.
0 * * * *
dfs.mover.auto.hdfsfiles_or_dirs
Specifies HDFS file and directory lists that implement automatic replica migration in specified clusters. Separate the lists with spaces.
This parameter is valid only when dfs.mover.auto.enable is set to true.
-
- Click Save to make configurations take effect. You do not need to restart the HDFS service.
Manually Migrating Data
- Install the client. If the client has been installed, skip this step.
For example, the installation directory is /opt/client. You need to change it to the actual installation directory.
For details about how to download and install the cluster client, see Installing an MRS Cluster Client.
- Log in to the node where the client is installed as the client installation user.
- Go to the client installation directory, for example, /opt/client.
cd /opt/client
- Run the following command to configure environment variables:
source bigdata_env
- If the cluster is in security mode, run the following command to authenticate the user. If the cluster is in normal mode, skip this step. The user must have the supergroup permission.
kinit Component service user
- Run the mover command in the following format:
hdfs mover -p <HDFS file or directory>
-p: specifies the HDFS file or directory from which data is to be migrated. You can specify a single file or a directory. If a directory is specified, all files in the directory will be migrated.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot