Creating a Batch Object Storage Migration Workflow

This section describes how to create a workflow to efficiently migrate buckets in batches.

Constraints

For more information, see What Are the Restrictions on Using MgC for Storage Migration?
Data in the Deep Cold Archive storage of Alibaba Cloud OSS cannot be restored using the restoration function of MgC. You need to manually restore the data before migration.
Microsoft Azure Blob does not support automatic archiving of restored data. If needed, you can manually archive the data after the migration is complete.

Warnings

When you create a workflow, there are three options for Overwrite Existing: Never, Always, and If older or different size. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.

Prerequisites

A batch object storage migration plan has been created for the source buckets to be migrated.

Procedure

Sign in to the MgC console. In the navigation pane, under Project, select an application migration project from the drop-down list.
In the navigation pane, choose Workflows.
Click Create Workflow in the upper right corner of the page.
In the Batch Object Storage Migration card, click Preview Steps to view the stages and steps predefined in the template and the detailed description of each stage and step. Automated steps are handled by the system. Click Configure Workflow in the lower right corner.
In the Basic Information area, enter a name and description for the workflow.
In the Migration Plan area, select the created migration plan. Then you should view the overview of the migration plan. Click View Details to view more information about the plan.
In the Migration Cluster area, select the cluster used for the migration. The cluster specified in the migration plan is preselected by default, but you can select another one if needed. The modification is only applied to the current workflow, not the migration plan.

Configure migration settings based on Table 1.

**Table 1** Migration settings
Parameter	Sub-parameter	Description
Concurrent Subtasks	-	User-defined. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, the maximum number of subtasks can be 20 or any number below.
Overwrite Existing	Never	Files existing at the target will never be overwritten. WARNING: If you choose Never for the initial migration, the attributes of involved parent folders at the source will not be migrated to the target. As a result, the folder attributes may be incomplete at the target. To avoid this issue, use the Never option with caution for the initial migration. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.
	Always	Files existing at the target will always be overwritten.
	If older or different size	The system replaces existing target files if they are older than or differ in size from their source counterparts. Files with matching modification times and sizes remain unchanged and are skipped from migration. The system verifies folders after their content is migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source. NOTE: The same overwriting policy is applied to empty folders as files.
	If different CRC 64 checksum	If a source object has a CRC64 checksum different from the paired target object, the source object will overwrite the target one. Otherwise, the source object will be skipped during the migration. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked. NOTE: This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud. Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
Consistency Verification	Size and last modified	With this default method, the system checks data consistency by comparing object size and last modification time.
Consistency Verification	CRC64 checksum	The system checks data consistency by comparing the CRC64 values in the metadata of source and target objects. If a source object or the paired target object does not have a CRC64 checksum, the OMS-calculated CRC64 value is used for verification. CRC64 verification may generate extra public traffic and request costs. For details, see Consistency Verification. NOTE: This option is only available for migration within Huawei Cloud or from AWS, Alibaba Cloud, or Tencent Cloud. This option is only available for migration from NAS_NFS_V3_MOUNT and NAS_NFS_V3_PROTOCOL. Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
Migrate Metadata	-	Determine whether to migrate metadata. If you select this option, object metadata will be migrated. If you do not select this option, only the Content-Type and Content-Encoding metadata will be migrated.

(Optional) Configure advanced options based on Table 2.

**Table 2** Advanced options
Parameter	Description
Migrate Incremental Data	If you select No, incremental migration will not be performed.
Migrate Incremental Data	If you select Yes, configure the overwriting policy and the execution mode for incremental migration. For details, see Configuring Incremental Migration Settings.
Target Storage Class	Choose the storage class that your data will be migrated to. For details about storage classes, see How Do I Choose Storage Classes? NOTE: CRC64-based consistency verification is not available for Archive and Deep Archive storage classes. Even if you choose the CRC64-based verification method, the system automatically uses the object size and last modification time to verify data consistency.
Enable KMS Encryption	If you do not select this option, whether migrated data will be encrypted in the target buckets depends on the server-side encryption setting of the buckets. If you select this option, all migrated objects will be encrypted before they are stored to the target bucket. NOTE: Using KMS to encrypt migrated data may slow down the migration speed by about 10%. This option is only available when KMS is supported in the region you are migrating to.
Restore Archive Data	If you do not select this option, the system records archived objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. If you select this option, the system automatically restores and migrates archived objects in the migration task. If an archived object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. NOTE: The system will restore all listed archived data before migrating it, and you pay the source cloud platform for the API requests and storage space generated accordingly. Data in the Deep Cold Archive storage of Alibaba Cloud OSS cannot be restored using the restoration function of MgC. You need to manually restore the data before migration. Microsoft Azure Blob does not support automatic archiving of restored data. If needed, you can manually archive the data after the migration is complete.
Filter Source Data	Filter files to be migrated by applying filters. For details about the filters, see Source Data Filters.
Download Data from CDN	If the default domain name cannot meet your migration requirements, then as long as the source cloud service provider supports custom domain names, you can bind a custom domain name to the source bucket, and enable the CDN service on the source platform to reduce data download expenses. Enter a custom domain name in the Domain Name text box and select a transmission protocol. HTTPS is more secure than HTTP and is recommended. If the migration source is the Alibaba Cloud OSS or Tencent Cloud COS, you also need to select an authentication type and enter an authentication key.
Send SMN Notification	Determine whether to use SMN to get notifications about migration results. If you do not select this option, no SMN messages will be sent after the migration is complete. If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.
Limit Traffic	Set the maximum bandwidth to be used by the migration workflow during different periods. If you do not select this option, migration traffic is not limited. If you select this option, limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited outside this period. NOTE: The bandwidth limit ranges from 1 MB/s to 1,048,576 MB/s. A maximum of five rules can be added. The time is the local standard time of the region you are migrating to.

Click Next: Confirm.
Confirm the workflow settings and click Confirm. The Run Workflow dialog box is displayed, which indicates that the workflow has been created.
- If you want to start the migration immediately, click Confirm to run the workflow.
- If you want to add a stage or step to the workflow, click Cancel. The workflow enters a Waiting state, and the migration has not started yet. To start the migration, click Run in the Operation column.
On the migration workflow details page, view the workflow settings and the migration progress. You can also perform the following operations:
- Move the cursor to the migration progress bar of a resource. In the displayed window, view the migration details about the resource.
- When a migration reaches a step that requires manual confirmation, place the cursor on the progress bar and click Confirm next to the step status in the displayed window. The migration can continue only after you confirm.
- In the Basic Information area, click Manage next to the cluster name. The cluster details page is displayed on the right. On the displayed page, you can:
  - Add, edit, or delete traffic limiting rules to control cluster traffic based on your requirements.
  - Add or delete migration nodes or list nodes, or upgrade plug-ins for existing nodes as required.
- In the Basic Information area, expand Advanced Settings. Review the incremental migration settings. If Incremental Migration Method is set to Automated, you can modify the number of incremental migrations.

Configuring Incremental Migration Settings

The following table describes the parameters about incremental migration.

**Table 3** Configuring incremental migration settings
Parameter	Description
Overwrite Existing During Incremental Migration	See Table 1. NOTE: This setting only applies to incremental migration.
Incremental Migration Method	Manual: You can manually initiate an incremental migration after the first incremental migration defined in the workflow is complete. Automated: You can define the interval, maximum attempts, and time zone to trigger incremental migrations. For example, if you configure the incremental migration settings as shown in the following figure, the system will initiate incremental migration at 8:00 in the morning on Mondays (GMT+08:00) for the next two weeks.

If you select Manual, incremental migration will be executed immediately when the workflow reaches the BatchCreateIncrementalMigrationTasks step.
If you select Automated, when the workflow reaches the BatchCreateIncrementalMigrationTasks step, you can:
- Manually trigger incremental migration immediately when you are waiting for automatic incremental migration.
- Skip incremental migration and verify services.