Migrating Data from Multiple Source Buckets by Prefix

Preparations

• Prepare a HUAWEI ID or an IAM user that has the permissions to access MgC. For details, see Preparations.
• Create an application migration project on the MgC console.
• Add the AK/SK pair of the account that owns the source buckets and the AK/SK pair of the account that owns the target buckets to MgC. For details, see Managing Credentials. Select AK/SK as the authentication mode. The source AK/SK pair is used to collect data from the source buckets, and the target AK/SK pair is used to migrate data to the target buckets.
• Ensure that the source and target accounts have the permissions required for the migration. For details, see How Do I Obtain Required Permissions for the Source and Target Accounts?
• On Huawei Cloud, create OBS buckets for receiving migrated data. For details, see Creating a Bucket. You can also use existing buckets.
• Prepare prefix lists.

  Create a prefix list for each source bucket to be migrated. Each prefix list must meet the following requirements:

  • The file must be in .txt format and must not exceed 2 MB in size.
  • Each line should contain only one prefix, with a maximum length of 1,024 characters per prefix.
  • A single file can contain up to 1,000 prefixes.

Precautions

• Supported regions
  MgC allows you to migrate object data in batches to the regions below. To migrate to other regions, use RDA or other solutions.

  • LA-Santiago
  • LA-Sao Paulo
  • TR-Istanbul
  • AP-Bangkok
  • AP-Singapore
  • AP-Jakarta
  • ME-Riyadh
  • CN North-Beijing4
  • CN East-Shanghai1
• Migration over intranets

  Data can be migrated between buckets in the same region over the intranet.

• Symbolic link processing
  MgC cannot migrate symbolic links. To migrate a path pointed to by a symbolic link, you need to:

  • Enter the actual path when specifying the migration path.
  • After the migration is complete, manually create a symbolic link to the path at the target.

Step 1: Discovering Source Buckets

1. Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
2. In the navigation pane, choose Source Resources.
3. Under Online Discovery, click Cloud Discovery.
   Figure 1 Cloud platform discovery
   

4. Set parameters in the Task Settings and Basic Information areas based on Table 1.

   Table 1 Parameters in the Task Settings and Basic Information areas

   Area	Parameter	Description
   Task Settings	Source Platform	Select the source cloud platform. In this example, select Huawei Cloud.
   	Credential	Select the source credential added in Preparations. If you did not add the credential, choose Add Credential. In the displayed area, set Authentication to AK/SK, enter the AK/SK pair of the source account, and click Verify and Save.
   	Region	Select the region where the source resources are located. You can select multiple regions.
   	Resource Type	Choose Storage > Object Storage from the drop-down list.
   Basic Information	Task Name	The default name format is <task type>-<current-time>. You can also specify a name.
   	Task Description (Optional)	Describe the task.
   	Application (Optional)	Select the application that you want to group the discovered resources into. If no applications are available, perform the following steps to create one: Click Create Application. Enter an application name and description. Configure Business Scenario and Environment, and set Region to the region that you want to migrate to. Click OK.

5. Click Confirm. After the task for discovering object storage resources is created, the system automatically starts collecting resource details.

   When the task status changes to Succeeded, the collection is completed.

Step 2: Create a Migration Cluster

Migration clusters incur addtional costs. For details, see Billing.

To ensure migration stability and data security, you are not allowed to log in to nodes in migration clusters. If you indeed need to log in to the nodes, contact technical support.

1. In the navigation pane on the left, choose Other > Migration Clusters.
2. Click Create Cluster in the upper right corner of the page.

   If this is your first time to create a migration cluster, you need to delegate MgC the required permissions. Click to view the permissions to be assigned.

   

3. Configure the parameters based on Table 2.

   Table 2 Parameters for creating a cluster

   Area	Parameter	Configuration	Constraints
   Basic Settings	Cluster Name	User-defined	The cluster name must be unique in the same account.
   	Region	Select the region to provision the cluster.	The cluster must be provisioned in the target region you are migrating to.
   	Cluster Type	Select what the cluster will be used for.	Currently, only storage migration is supported.
   Node Settings	Key Pair	It is used for identity authentication when you remotely log in to the nodes (ECSs) in the cluster. You need to create a key pair in advance. For details, see Creating a Key Pair.	-
   	Master Node Specifications	It is used to manage migration nodes and list nodes.	A cluster can only have one master node.
   	Migration Node Specifications	Migration nodes are used for executing migration and verification tasks. The recommended specifications are 8 vCPUs and 16 GB of memory.	The node specifications cannot be modified after the cluster is created. The number of nodes must meet the following requirements: Number of migration nodes + Number of list nodes + 1 ≤ 100 Number of migration nodes + Number of list nodes + 1 ≤ Number of unused IP addresses in the subnet
   	List Node Specifications	List nodes are used for listing tasks. The recommended specifications are 8 vCPUs and 16 GB of memory.
   Network Settings	VPC	Select a VPC from the drop-down list.	-
   	Subnet	Make sure that there are enough unused IP addresses for the migration and list nodes in this cluster.	Number of unused IP addresses in the subnet ≥ Number of migration nodes + Number of list nodes + 1
   	Network Type	Internet: You need to select a public NAT gateway. If there is no gateway available, choose Buy Gateway from the drop-down list and select the gateway specifications and EIPs you want to associate with the gateway. A maximum of 20 EIPs can be selected at a time. Intranet: This option is suitable for data migration within a region. Private line: Source data is directly accessed through the private line. For details about how to set up a Direct Connect connection, see Direct Connect Best Practices.	-
   Advanced Settings	DNS Configuration (Optional)	Enter the IP address of the DNS server to update the value of nameserver in the /etc/resolv.conf file. Use commas (,) to separate multiple DNS server addresses, for example, 192.0.2.1,192.0.2.2.	A maximum of three DNS IP addresses can be specified.
   	Domain Mapping (Optional)	Add mappings between domain names and IP addresses to update the /etc/hosts file.	A maximum of 500 mappings can be added.
   -	Traffic Limiting	Set the maximum bandwidth for the migration task during different periods. If you do not select this option, migration traffic will not be limited. If you select this option, you can limit the migration traffic by setting the start time, end time, and bandwidth limit. NOTICE: For example, if you set Start Time to 08:00, End Time to 12:00, and Maximum Bandwidth to 20 MB/s, the maximum migration speed is limited to 20 MB/s when the migration task is running in the period from 08:00 to 12:00. The migration speed is not limited outside this period.	A maximum of five traffic limiting rules can be added. The time is the local standard time of the region you are migrating to.
   	Log Collection	If this option is enabled, storage migration logs will be collected for possible troubleshooting later. If this option is disabled, storage migration logs will not be collected.	-

4. Click Confirm. Then you can view the cluster in the list. For details about cluster statuses, see Cluster Statuses. If the cluster status is Creation failed, move the cursor to the status to view the failure cause. After the fault is rectified, choose More > Retry to create the cluster again.

Step 3: Create a Batch Object Storage Migration Plan

1. Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
2. In the navigation pane, choose Migration Plans (NEW).
3. Click Create Migration Plan.
4. Set migration plan parameters based on Table 3.

   Table 3 Parameters for creating a migration plan

   Parameter	Configuration
   Migration Plan Template	Select Batch Object Storage Migration.
   Plan Name	The system assigns a default plan name. You can also specify a plan name.
   Description (Optional)	Enter a description.

5. Click Confirm. The migration plan for batch bucket migration is created, and the page for configuring source and target resources is displayed.

Step 4: Configure Migration Resources

1. On the Configure Source/Target page, select the source cloud platform from the Source Platform drop-down list. A migration plan can be used to migrate buckets from the same source platform and to the same target region.
2. Select the region where the target buckets are located from the Target Region drop-down list.
3. Above the source bucket list, click Add.
4. Select the buckets to be migrated, click Modify in the Operation column, configure endpoints, and click Confirm. A maximum of 20 buckets can be added.
5. In the Buckets area, set the bucket type, source credential, target bucket, and target credential. After the settings are complete, click Save in the Operation column. You can also click Save All after setting multiple buckets. You can modify the settings after saving them.
6. After you configure all the buckets and save the settings, click Next.

Step 5: Configure Migration Settings

1. In the Migration Settings area, set the migration method, migration scope, and (optional) target prefix based on Table 4.

   Table 4 Migration methods and scope

   Migration Method	Description	Migration Scope	Target Prefix (Optional)
   Prefix migration	Migrates objects matched with the specified names or prefixes.	There are two methods to specify source prefixes. If no source prefixes are specified, all objects will be migrated. Import Source Prefix: Import the source prefix lists prepared. Add Source Prefix: If no prefix lists are preprared, manually add prefixes.	For each migration, only one target prefix can be added to organize objects migrated to the target bucket. The prefix cannot start with a slash (/). For example, if you specify the prefix D, the source file /A/B/C.txt will be replaced to /D/A/B/C.txt in the target bucket.

2. (Optional) In the Source Settings area, choose whether to enable Migrate Metadata and Restore Archive Data.
   

   If the source is Alibaba Cloud OSS or Tencent Cloud COS, you can enable Download Data from CDN. If CDN authentication is enabled, you also need to select an authentication type and enter an authentication key.

3. In the Target Settings area, set parameters according to Table 5.

   Table 5 Target settings

   Parameter	Sub-parameter	Description
   Overwrite Existing	If older or different size	The system replaces existing target files if they are older than or differ in size from their source counterparts. Files with matching modification times and sizes remain unchanged and are skipped from migration. The system verifies folders after their content is migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source.
   	Never	Files existing at the target will never be overwritten. WARNING: If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.
   	Always	Files existing at the target will always be overwritten.
   	If different CRC 64 checksum	This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud. If a source object has a CRC64 checksum different from the paired target object, the source object will overwrite the target one. Otherwise, the source object will be skipped during the migration. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked.
   Target Storage Class	Standard	For details about storage classes, see How Do I Choose Storage Classes? NOTE: CRC64-based consistency verification is not available for Archive and Deep Archive storage classes. Even if you choose the CRC64-based verification method, the system automatically uses the object size and last modification time to verify data consistency.
   	Infrequent Access
   	Archive
   	Deep Archive
   	Same as source

4. In the Consistency Verification area, select a consistency check method.
   • Size and last modified: With this default setting, the system verifies object consistency with object size and last modification time.
   • CRC64 checksum: This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud. The system verifies data consistency by comparing CRC64 values in the metadata. If a source object or the paired target object does not have a CRC64 checksum, their sizes and last modification times are verified.

5. In the Migration Settings area, set parameters according to Table 6.

   Table 6 Migration settings

   Parameter	Description
   Concurrent Subtasks	User-defined. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, the maximum number of subtasks can be 20 or any number below.
   Limit Traffic	Set the maximum bandwidth for the migration task during different periods. If you do not select this option, migration traffic will not be limited. If you select this option, you can limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited outside this period. NOTE: The bandwidth limit ranges from 1 MB/s to 1,048,576 MB/s. A maximum of five rules can be added. The time is the local standard time of the region you are migrating to.

6. (Optional) Configure advanced options based on Table 7.

   Table 7 Advanced options

   Parameter	Description
   Enable KMS Encryption	If you do not select this option, whether migrated data will be encrypted in the target buckets depends on the server-side encryption setting of the buckets. If you select this option, all migrated objects will be encrypted before they are stored to the target bucket. NOTE: Using KMS to encrypt migrated data may slow down the migration speed by about 10%. This option is only available when KMS is supported in the region you are migrating to.
   Filter Source Data	Use exclude patterns, include patterns, and time ranges to filter files to be migrated. For details about filters, see Source Data Filters.
   Send SMN Notifications	Determine whether to use SMN to get notifications about migration results. If you do not select this option, no SMN messages will be sent after the migration is complete. If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.
   Migrate Incremental Data	If this option is not selected, incremental migration will not be performed. If this option is selected, configure the overwriting policy and specify how to execute incremental migration. For details, see Configuring Incremental Migration Settings.

7. After the migration settings are configured, click Next to open the Configure Migration Cluster page.

Step 6: Configure a Migration Cluster

1. In the Configuration Mode area, select Use existing cluster.
2. In the Migration Cluster area, select the migration cluster created in step 2. To modify the cluster, click Modify in the Operation column and modify the cluster by referring to Managing a Migration Cluster.
3. After the migration cluster is configured, click Next to open the confirmation page.

Step 7: Create a Batch Object Storage Migration Workflow

1. On the Confirm page, check the resource and migration settings carefully to ensure that they meet your requirements. To update the settings, click Previous to return to the corresponding stage for modification.
2. After confirming the settings are correct, click Create Workflow in the lower right corner of the page. The Start Workflow dialog box is displayed, indicating that the workflow has been created.
   • If you want to start the migration immediately, click OK to start the workflow and view the workflow details.
   • If you want to save the migration workflow without starting the migration, click Cancel. The migration workflow will move to a Waiting status, and the migration has not started yet. To run the workflow, click Run in the Operation column to manually start the migration.

   For more information about migration workflows, see Creating a Batch Object Storage Migration Workflow.