Migrating Data from Multiple Source Buckets by Prefix
This section describes how to filter objects for migration in source buckets using prefixes and migrate the objects to Huawei Cloud OBS buckets.
Preparations
- Prepare a Huawei Cloud account or an IAM user that has the permissions to access MgC. For details, see Preparations.
- Create an application migration project on the MgC console.
- Add the AK/SK pair of the account that owns the source buckets and the AK/SK pair of the account that owns the target buckets to MgC. For details, see Managing Credentials. Select AK/SK as the authentication mode. The source AK/SK pair is used to collect data from the source buckets, and the target AK/SK pair is used to migrate data to the target buckets.
- Ensure that the source and target accounts have the permissions required for the migration. For details, see How Do I Obtain Required Permissions for the Source and Target Accounts?
- On Huawei Cloud, create OBS buckets for receiving migrated data. For details, see Creating a Bucket. You can also use existing buckets.
- Prepare prefix lists.
Create a prefix list for each source bucket to be migrated. Each prefix list must meet the following requirements:
- The file must be in .txt format and must not exceed 2 MB in size.
- Each line should contain only one prefix, with a maximum length of 1,024 characters per prefix.
- A single file can contain up to 1,000 prefixes.
Precautions
- Supported regions MgC allows you to migrate object data in batches to the regions below. To migrate to other regions, use RDA or other solutions.
- LA-Santiago
- LA-Sao Paulo
- TR-Istanbul
- AP-Bangkok
- AP-Singapore
- AP-Jakarta
- ME-Riyadh
- CN North-Beijing4
- CN East-Shanghai1
- Migration over intranets
Data can be migrated between buckets in the same region over the intranet.
- Symbolic link processing MgC cannot migrate symbolic links. To migrate a path pointed to by a symbolic link, you need to:
- Enter the actual path when specifying the migration path.
- After the migration is complete, manually create a symbolic link to the path at the target.
Step 1: Discovering Source Buckets
- Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
- In the navigation pane, click Resource Survey.
- Under Online Discovery, click Cloud Discovery. Figure 1 Cloud discovery
- Set parameters in the Task Settings and Basic Information areas based on Table 1.
Table 1 Parameters in the Task Settings and Basic Information areas Area
Parameter
Description
Task Settings
Source Platform
Select the source cloud platform. In this example, select Huawei Cloud.
Credential
Select the source credential added in Preparations. If you did not add the credential, click Add Credential. In the displayed area, set Authentication to AK/SK, enter the AK/SK pair of the source account, and save the changes.
Region
Select the region where the source resources are located. You can select multiple regions.
Resource Type
Choose Storage > Object Storage from the drop-down list.
Basic Information
Task Name
The default name format is <task type>-<current-time>. You can also specify a name.
Task Description (Optional)
Describe the task.
Application (Optional)
Select the application that you want to group the discovered resources into. If no applications are available, perform the following steps to create one:
- Click Create Application. Enter an application name and description.
- Configure Business Scenario and Environment, and set Region to the region that you want to migrate to.
- Click OK.
- Click Confirm. After the task for discovering object storage resources is created, the system automatically starts collecting resource details.
When the task status changes to Succeeded, the collection is completed.
Step 2: Create a Migration Cluster
Migration clusters incur additional costs. For details, see Billing.
To ensure migration stability and data security, you are not allowed to log in to nodes in migration clusters. If you indeed need to log in to the nodes, contact technical support.
- In the navigation pane on the left, choose Other > Migration Clusters.
- Click Create Cluster in the upper right corner of the page.
If this is your first time creating a migration cluster, you need to assign MgC the required permissions. Click
to view the permissions to be assigned.
- Configure the parameters based on Table 2.
Table 2 Parameters for creating a cluster Area
Parameter
Configuration
Constraints
Basic Settings
Cluster Name
User-defined
The cluster name must be unique in the same account.
Region
Select the region to provision the cluster.
The cluster must be provisioned in the target region you are migrating to.
Cluster Type
Select what the cluster will be used for.
Currently, only storage migration is supported.
Node Settings
Key Pair
It is used for identity authentication when you remotely log in to the nodes (ECSs) in the cluster. You need to create a key pair in advance. For details, see Creating a Key Pair.
-
Enterprise Project
This function is provided for enterprise users. You can add the ECS to an enterprise project. If no enterprise projects are available, the system creates the default enterprise project.
An enterprise project facilitates project-level management and grouping of cloud resources and users. The default project is default.
For details about how to create and manage enterprise projects, see Enterprise Management User Guide.
-
Master Node Specifications
It is used to manage migration nodes and list nodes.
A cluster can only have one master node.
Migration Node Specifications
Migration nodes are used for executing migration and verification tasks. The recommended specifications are 8 vCPUs and 16 GB of memory.
- The node specifications cannot be modified after the cluster is created.
- The number of nodes must meet the following requirements:
- Number of migration nodes + Number of list nodes + 1 ≤ 100
- Number of migration nodes + Number of list nodes + 1 ≤ Number of unused IP addresses in the subnet
List Node Specifications
List nodes are used for listing tasks. The recommended specifications are 8 vCPUs and 16 GB of memory.
Network Settings
VPC
Select a VPC from the drop-down list.
-
Subnet
Make sure that there are enough unused IP addresses for the migration and list nodes in this cluster.
Number of unused IP addresses in the subnet ≥ Number of migration nodes + Number of list nodes + 1
Network Type
- Internet: You need to select a public NAT gateway. If there is no gateway available, choose Buy Gateway from the drop-down list and select the gateway specifications and EIPs you want to associate with the gateway. A maximum of 20 EIPs can be selected at a time.
- Intranet: This option is suitable for data migration within a region.
- Private line: Source data is directly accessed through the private line. For details about how to set up a Direct Connect connection, see Direct Connect Best Practices.
-
Advanced Settings
DNS Configuration (Optional)
Enter the IP address of the DNS server to update the value of nameserver in the /etc/resolv.conf file. Use commas (,) to separate multiple DNS server addresses, for example, 192.0.2.1,192.0.2.2.
A maximum of three DNS IP addresses can be specified.
Domain Mapping (Optional)
Add mappings between domain names and IP addresses to update the /etc/hosts file.
A maximum of 500 mappings can be added.
-
Limit Traffic
Set the maximum bandwidth for the migration task during different periods.
- If you do not select this option, migration traffic will not be limited.
- Selected: Traffic is controlled by setting the start time, end time, and bandwidth limit. NOTICE:
For example, if you set Start Time to 08:00, End Time to 12:00, and Maximum Bandwidth to 20 MB/s, the maximum migration speed is limited to 20 MB/s when the migration task is running in the period from 08:00 to 12:00. The migration speed is not limited beyond this period.
- A maximum of five traffic limiting rules can be added.
- The time is the local standard time of the region you selected.
Log Collection
- If this option is enabled, storage migration logs will be collected for possible troubleshooting later.
- If this option is disabled, storage migration logs will not be collected.
-
- Click Confirm. Then you can view the cluster in the list. For details about cluster statuses, see Cluster Statuses. If the cluster status is Creation failed, move the cursor to the status to view the failure cause. After the fault is rectified, choose More > Retry to create the cluster again.
Step 3: Create a Batch Object Storage Migration Task
- In the navigation pane on the left, choose Storage Migration > Batch Storage Migration.
- Choose Object Storage > Create Migration Task.
- Use the default task name or create one. Enter the task description (optional).
- Click OK to complete the task creation.
Step 4: Configure the Migration Task
This step includes the following phases:
Step 5: Track the Migration Task
- Checking the resource status: Click a resource name. On the Resource Details page displayed on the right, you can review the basic information, migration status, and migration progress of the resource.
- Confirming steps: Some resources require manual confirmation. In this case, click Confirm in the Migration Status column or in the Operation column to continue the migration.
To check the migration phases and steps included in a migration task, click View Task Template in the upper right corner of the task details page.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot