Creating a Batch Object Storage Migration Plan
You can configure migration plans using templates designed for different migration scenarios. Then you can create migration workflows on the top of these plans. Follow this section to configure a migration plan for batch migrating object storage buckets.
The Batch Object Storage Migration template is designed to migrate buckets in batches. If only one bucket needs to be migrated, create a migration workflow directly.
Prerequisites
- Source buckets have been discovered over the Internet and their details have been collected. For details, see Cloud Discovery.
- (Optional) A deep collection has been performed for the buckets to be migrated.
Warnings
When creating a workflow, there are four overwriting policies for objects with the same name at the source and target: If older or different size, Never, Always, and If different CRC64 checksum. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.
Constraints
- Migration plans can be created only in the AP-Singapore and TR-Istanbul regions.
- A migration plan can be used to migrate buckets from the same source platform and to the same target region.
- A source bucket can be included in multiple migration plans.
- You can define the migration scope for a source bucket using multiple prefixes, but only one prefix can be specified to control data replacement in the target bucket.
- Each user is allowed to migrate a maximum of 20 buckets at the same time in a single migration cluster.
Step 1: Create a Migration Plan
- Sign in to the MgC console. In the navigation pane, under Project, select an application migration project from the drop-down list.
- In the navigation pane, choose Migration Plans (NEW).
- Click Create Migration Plan.
- Set migration plan parameters based on Table 1.
- Click Confirm. The migration plan for batch bucket migration is created, and the page for configuring source and target resources is displayed.
Step 2: Configure Migration Resources
- On the Configure Source/Target page, select the source cloud platform from the Source Platform drop-down list. A migration plan can be used to migrate buckets from the same source platform and to the same target region.
- Select the region where the target buckets are located from the Target Region drop-down list.
- Above the source bucket list, click Add.
- Select the buckets to be migrated, click Modify in the Operation column, configure endpoints, and click Confirm. A maximum of 100 buckets can be added.
- In the Buckets area, set the bucket type, source credential, target bucket, and target credential. After the settings are complete, click Save in the Operation column. You can also click Save All after setting multiple buckets. You can modify the settings after saving them.
- After you configure all the buckets, click Next.
Step 3: Configure Migration Settings
- In the Migration Settings area, set the migration method, migration scope, and (optional) target prefix. Refer to Table 2 to learn how to configure Migration Method and Migration Scope.
Table 2 Migration methods and scope Migration Method
Description
Migration Scope
Full migration
Migrates all data in a source bucket or in specific paths.
N/A
Prefix migration
Migrates objects matched with the specified names or prefixes.
There are two methods to specify source prefixes. If no source prefix is specified, all objects are migrated.
- Import Source Prefixes: Import a TXT file that lists prefixes.
- Add Source Prefixes: Enter prefixes.
List migration
Migrates objects recorded in the list files. The list files must be stored in the target bucket.
(Mandatory) Enter the path for storing the list files in the Migration Scope box and end the path with a slash (/).
- (Optional) In the Source Settings area, select Migrate Metadata and Restore Archive Data.
If the source is Alibaba Cloud OSS or Tencent Cloud COS, you can enable Download Data from CDN. If CDN authentication is enabled, you also need to select an authentication type and enter an authentication key.
- In the Target Settings area, set parameters according to Table 3.
Table 3 Target settings Parameter
Sub-parameter
Description
Overwrite Existing
If older or different size
- The system replaces existing target files if they are older than or differ in size from their source counterparts. Files with matching modification times and sizes remain unchanged and are skipped from migration.
- The system verifies folders after their content is migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source.
Never
Files existing at the target will never be overwritten.
WARNING:If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.
Always
Files existing at the target will always be overwritten.
If different CRC 64 checksum
If a source object has a CRC64 checksum different from the paired target object, the source object will overwrite the target one. Otherwise, the source object will be skipped during the migration. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked.
NOTE:- This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud.
- Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
Target Storage Class
Standard
For details about storage classes, see How Do I Choose Storage Classes?
NOTE:CRC64-based consistency verification is not available for Archive and Deep Archive storage classes. Even if you choose the CRC64-based verification method, the system automatically uses the object size and last modification time to verify data consistency.
Infrequent Access
Archive
Deep Archive
Same as source
- In the Consistency Verification area, select a consistency check method.
- Size and last modified: With this default setting, the system verifies object consistency with object size and last modification time.
- CRC64 checksum: The system checks data consistency by comparing the CRC64 values in the metadata of source and target objects. If a source object or the paired target object does not have a CRC64 checksum, the OMS-calculated CRC64 value is used for verification. CRC64 verification may generate extra public traffic and request costs. For details, see Consistency Verification.
- This option is only available for migration within Huawei Cloud or from AWS, Alibaba Cloud, or Tencent Cloud.
- This option is only available for migration from NAS_NFS_V3_MOUNT and NAS_NFS_V3_PROTOCOL.
- Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
- In the Migration Settings area, set parameters according to Table 4.
Table 4 Migration settings Parameter
Description
Concurrent Subtasks
User-defined. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, the maximum number of subtasks can be 20 or any number below.
Limit Traffic
Set the maximum bandwidth to be used by the migration workflow during different periods.
- If you do not select this option, migration traffic is not limited.
- If you select this option, limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit.
For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited outside this period.
NOTE:- The bandwidth limit ranges from 1 MB/s to 1,048,576 MB/s.
- A maximum of five rules can be added.
- The time is the local standard time of the region you are migrating to.
- (Optional) Configure advanced options based on Table 5.
Table 5 Advanced options Parameter
Description
Enable KMS Encryption
- If you do not select this option, whether migrated data will be encrypted in the target buckets depends on the server-side encryption setting of the buckets.
- If you select this option, all migrated objects will be encrypted before they are stored to the target bucket.
NOTE:- Using KMS to encrypt migrated data may slow down the migration speed by about 10%.
- This option is only available when KMS is supported in the region you are migrating to.
Filter Source Data
Filter files to be migrated by applying filters. For details about the filters, see Source Data Filters.
Send SMN Notifications
Determine whether to use SMN to get notifications about migration results.
- If you do not select this option, no SMN messages will be sent after the migration is complete.
- If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.
Migrating Incremental Data
- If this option is not selected, incremental migration is not performed.
- If this option is selected, configure the overwriting policy and the execution mode for incremental migration. For details, see Configuring Incremental Migration Settings.
- After the migration settings are configured, click Next to open the Configure Migration Cluster page.
Step 4: Configure a Migration Cluster
- In the Configuration Method area, select a method of configuring a migration cluster. If a cluster is available, choose Use existing cluster. If no migration clusters are available, choose Explore recommendations and create new cluster to assess the appropriate cluster size. You can assess how large of a migration cluster is required for the migration and then create a migration cluster with the recommended specifications. Alternatively, you can skip this step and use an existing migration cluster. For details about how to create a cluster, see Creating a Migration Cluster.
- In the Migration Cluster area, select the cluster to be used and click OK. If you want to modify an existing cluster to meet your requirements, see Managing a Migration Cluster.
- After the migration cluster is configured, click Next to open the confirmation page.
Step 5: Confirm the Configurations
After performing steps 1 to 4, check the resource and migration settings on the Confirm page. To update the settings, click Previous to return to the corresponding stage for modification.
Step 6: Perform Follow-Up Operations
After confirming that the configurations are correct, perform follow-up operations as required.
- Click Design in the Operation column to update the migration plan.
- Click Create Workflow in the Operation column to create a migration workflow on the top of the plan. For details, see Creating a Batch Object Storage Migration Workflow.
- If you want to start the migration immediately, click OK to start the workflow and view the workflow details.
- If you want to save the migration workflow without starting the migration, click Cancel. The migration workflow will move to a Waiting status, and the migration has not started yet. To run the workflow, click Run in the Operation column to manually start the migration.
For details about migration workflows, see Creating a Batch Object Storage Migration Workflow.
Configuring Incremental Migration Settings
The following table describes the parameters about incremental migration.
Parameter |
Description |
---|---|
Overwrite Existing During Incremental Migration |
See Table 3.
NOTE:
This setting only applies to incremental migration. |
Incremental Migration Method |
|
- If you select Manual, incremental migration will be executed immediately when the workflow reaches the BatchCreateIncrementalMigrationTasks step.
- If you select Automated, when the workflow reaches the BatchCreateIncrementalMigrationTasks step, you can:
- Manually trigger incremental migration immediately when you are waiting for automatic incremental migration.
- Skip incremental migration and verify services.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot