Migrating OBS Bucket Data to SFS Turbo Using MgC

Solution Overview

You can use MgC to migrate data to SFS Turbo file systems from the following data sources: Huawei Cloud OBS, Alibaba Cloud OSS, Baidu Cloud BOS, Tencent Cloud COS, Kingsoft Cloud KS3, Qiniu Cloud KODO, UCloud US3, Amazon S3, Microsoft Blob, NAS_GFS, NAS_SMB, NAS_NFS_V3_MOUNT, NAS_NFS_V3_PROTOCOL, and HTTP/HTTPS data sources.

This section describes how to use a template to create a file storage migration workflow to migrate data from Huawei Cloud OBS to SFS Turbo.

For more information about data migration operations between other data sources and SFS Turbo, see Creating a Storage Migration Workflow.

Constraints

See the constraints described in What Are the Restrictions on Using MgC for Storage Migration?

Warnings

When you create a workflow, there are three options for Overwrite Existing: Never, Always, and If older or different size. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even if the task may appear successful. This could impact data integrity, so use the Never option with caution.

Prerequisites

You have created an SFS Turbo file system and obtained its shared path. For how to create an SFS Turbo file system, see Creating an SFS Turbo File System. For how to mount an SFS Turbo file system, see Mounting an SFS Turbo File System.
You have created a migration cluster. For details, see Creating a Migration Cluster.

Resource Planning

Table 1 describes the resource planning in this solution.

**Table 1** Resource planning
Resource	Example Configuration
Huawei Cloud OBS bucket	Bucket Name: bucket Bucket Type: Bucket Region: CN-Hong Kong Endpoint: obs.ap-southeast-1.myhuaweicloud.com

Creating a File Storage Migration Workflow

Sign in to the . In the navigation pane, under Project, select your application migration project from the drop-down list.
In the navigation pane, choose Workflows.
Click Create Workflow in the upper right corner of the page.
In the Storage Migration card, click Preview Steps to view the stages and steps predefined in the template and the detailed description of each stage and step. Automated steps are handled by the system. Click Configure Workflow in the lower right corner.

Configure workflow parameters based on Table 2.

**Table 2** Workflow parameters
Parameter	Example Value	Description
Name	Data migration	User-defined.
Target Region	CN-Hong Kong	Select the region you are migrating to.
Cluster	cluster	Select a migration cluster. The cluster consists of a master node and several migration and list nodes. If no cluster is available, . NOTICE: The system automatically creates a master node in a migration cluster. You do not need to configure it.

Configure the migration source and target based on Table 3 and Table 4.

**Table 3** Parameters for configuring a migration source
Parameter	Example Value	Description
Location Type	Huawei Cloud OBS	Select the source storage type.
AK	-	Enter the AK of the source cloud account.
SK	-	Enter the SK of the source cloud account.
Bucket	bucket	Enter the name of the source bucket to be migrated.
Type	Bucket	Set this parameter based on the source bucket type.
Endpoint	obs.ap-southeast-1.myhuaweicloud.com	Enter the endpoint of the region where the source bucket is located.

**Table 4** Parameters for configuring a migration target
Parameter	Example Value	Description
Location Type	NAS_SMB	Select a Huawei Cloud storage type based on the source storage type. NAS_SMB is used for SMB file systems.
File System Address	192.1.1.1:/	Enter the shared path of the target SFS Turbo file system. You can click the icon next to the shared path on the file system list page to copy it.
Path	/	Enter the directory for storing migrated files. The format is /<directory-name>. If you enter /, the entire bucket data will be migrated.

Configure migration task parameters based on Table 5.

**Table 5** Migration task settings
Parameter	Example Value	Description
Task Type	Full migration	Full migration migrates all data in the source bucket.
Listing Mode	Serial	Serial is the default listing mode if the source is a bucket.
Concurrent Subtasks	-	User-defined. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, enter 20 or any number below.
Overwrite Existing	Always	Always indicates that files existing at the target will always be overwritten. WARNING: If you choose Never for the initial migration, the attributes of involved parent folders at the source will not be migrated to the target. As a result, the folder attributes may be incomplete at the target. To avoid this issue, use the Never option with caution for the initial migration. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even if the task may appear successful. This could impact data integrity, so use the Never option with caution.
Consistency Verification	Size and last modified	With this default method, the system checks data consistency by comparing object size and last modification time.
Clear Cluster	Select this option.	Determine whether to clear the migration cluster after the migration is complete. If you select this option, a step for clearing the migration cluster will be created in the workflow. You can also choose whether to clear resources used by the cluster, such as NAT gateways, security groups, and VPCEP resources. If you do not select this option, a step for clearing the migration cluster will not be created in the workflow, but the migration cluster and its resources will automatically be deleted 30 days after the workflow is created.

(Optional) Configure advanced settings based on Table 6.

**Table 6** Advanced options
Parameter	Example Value	Description
Restore Archive Data	Select this option.	If you do not select this option, the system records archived objects as objects failed to migrate and continues to migrate other objects in the migration task. If you select this option, the system automatically restores and migrates archived objects in the migration task. If an archived object fails to be restored, the system skips it and records it as an object failed to migrate and continues to migrate other objects in the migration task. NOTE: The system will restore archived data before migrating it, and you pay the source cloud platform for the API requests and storage space generated accordingly. Data in the Deep Cold Archive storage of Alibaba Cloud OSS cannot be restored by MgC. You need to manually restore data at the source before migration.
Filter Source Data	Select this option.	Use exclude patterns, include patterns, start time, and end time to filter files to be migrated. For details about filters, see Source Data Filters.
Send SMN Notifications	Select this option.	Determine whether to use SMN to get notifications about migration results. If you do not select this option, no SMN messages will be sent after the migration is complete. If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.
Limit Traffic	Select this option.	Set the maximum bandwidth to be used by the migration workflow during different periods. If you do not select this option, migration traffic is not limited. If you select this option, you can limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s when the migration task runs from 08:00 to 12:00. The migration speed is not limited outside this period. NOTE: The bandwidth limit ranges from 1 MB/s to 1048576 MB/s. A maximum of five rules can be added. The time is subject to the local standard time of the region you are migrating to.
Schedule Migration	Select this option.	Schedule the migration to run during a period. If you do not select this option, you need to manually start or stop the migration. If you select this option, the migration runs during the specified period and stops outside that period. For example: If you set Start Time to 08:00 and End Time to 12:00, the migration task runs from 08:00 to 12:00 every day. The migration stops outside that period. If you set Start Time to 12:00 and End Time to 08:00, the migration runs from 12:00 of the current day to 08:00 of the next day. The migration stops outside that period.

Click Next: Confirm.
Confirm the workflow settings, and click Confirm. After the Run Workflow dialog box is displayed, the workflow has been created.
- If you want to start the migration immediately, click Confirm to run the workflow.
- If you want to add a stage or step to the workflow, click Cancel. The workflow enters a Waiting state, and the migration has not started yet. To start the migration, click Run in the Operation column.