Migrating Object Storage Data Using a List
This section describes how to use MgC storage migration workflows to perform list-based migration.
Scenarios
List migration applies to the migration of over 100,000 objects from Alibaba Cloud Object Storage Service (OSS) to Huawei Cloud Object Storage Service (OBS).
Prerequisites
- You have signed up for a Huawei Cloud account and completed real-name authentication.
- You have obtained the AK/SK pairs for the source and target platform accounts.
- The source platform account needs the permissions required to list all buckets, obtain bucket locations, list objects, obtain object metadata, and obtain object content. For details about how to obtain these permissions, see Source Permissions.
- The target platform account needs the permissions required to list buckets, obtain bucket locations, list objects, obtain object metadata, obtain object content, upload objects, list uploaded parts, and restore archived objects. For details about how to obtain these permissions, see Target Permissions.
- You have buckets in OBS.
Step 1: Generate an Alibaba Cloud OSS Bucket List
- Sign in to the Alibaba Cloud OSS console and choose Buckets.
- On the Buckets page, click the name of the desired bucket.
- On the displayed page, choose Data Management > Bucket Inventory.
- On the Bucket Inventory page, click Create Inventory, enter a rule name, select an inventory storage bucket, and click OK.
- Return to the Bucket Inventory page and click View Inventory Lists in the Operation column. If a package exists in the data/ directory, the bucket inventory lists have been generated.

Step 2: Process Bucket Data
- Prepare a Linux VM with 4 vCPUs, 8 GB memory, and 100 GB disk space (depending on the data volume) on Alibaba Cloud. Bind an EIP to the VM.
- Log in to the VM as root.
- Run the following command to install zip:
yum install zip unzip
- Run the following command to install ossutil:
sudo -v ; curl https://gosspublic.alicdn.com/ossutil/install.sh | sudo bash
- Configure ossutil.
ossutil config # Set the configuration file path as prompted: # (1) Configuration file path # (2) The display language (EN/CH) # (3) AK SK
- Download the bucket inventory lists.
ossutil64 cp oss://<bucket-name>/<path>/data/ <CSV-file>/ --include "*.gz" -r # <bucket-name> indicates the name of the inventory storage bucket. # <path> indicates the path that stores the inventory lists. # <CSV-file> indicates the directory for storing downloaded CSV files. # --include "*.gz" indicates that files whose names contain .gz are downloaded. # -r indicates that all eligible files within recursive directories are downloaded.
- Run the following command to install gzip:
yum install gzip
- Run the following command to go to the directory where the package is stored:
cd <CSV-file> # <CSV-file> indicates the directory for storing the files downloaded in step 2.5.
- Run the following command to decompress the bucket inventory file package:
gzip -d *.csv.gz
- Select the corresponding CSV file and run the following command to extract object keys (URLs):
cat <CSV-name> | awk -F ",\"" '{print $2}' | awk -F "\"" '{print $1}' > <List-file> # <CSV-name> indicates the name of the file to be processed. # <List-file> indicates the TXT file where the extracted data is saved. - Split the list file. In the command below, xxxx.txt is the list file obtained in the previous step.
#!/bin/bash sourceFile="xxxx.txt" splitLine=100000 split -l $splitLine -d -a 5 --additional-suffix=.txt "$sourceFile" xxxx- mkdir xxxx-file mv xxxx-*.txt xxxx-file
Step 3: Upload the List to the Target OBS Bucket on Huawei Cloud
- Install obsutil on the VM prepared in step 2. For details, see Downloading and Installing obsutil.
- Initialize the configuration.
- Use obsutil to upload the lists.
./obsutil cp <file-path> obs:// <bucket-name> -f -r # <file-path> indicates the folder where the list files are stored. # <bucket-name> indicates the name of the target bucket.
Step 4: Create an Object Storage Migration Task
- Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
- In the navigation pane on the left, choose Storage Migration > Object Storage.
- Choose Exclusive Clusters > Create Migration Task.
- Set basic information based on Table 1.
Table 1 Basic parameters Parameter
Description
Name
User-defined. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed. The name must start with a letter or digit, and it must not start or end with a period.
Region
Select the region you are migrating to.
Description
User-defined. It can contain a maximum of 255 characters. Allowed characters include letters, digits, hyphens (-), underscores (_), commas (,), periods (.), spaces, and decimal points (.). It cannot start with a decimal point (.), hyphen (-), or underscore (_).
Cluster
Select a migration cluster. The cluster consists of a master node and several migration and list nodes. If there is no available cluster, create one by referring to Creating a Migration Cluster.
NOTE:A master node is created by the system in a migration cluster by default. You do not need to configure it.
- Configure the migration source and target based on Table 2 and Table 3.
Table 2 Parameters for configuring a migration source Parameter
Description
Remarks
Location Type
-
Enable KMS Encryption
Disabled: The source credential is not encrypted.
These parameters are available when cloud storage is selected for Location Type.
Enabled: The source credential is encrypted.
Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains:
- kms:cmk:get: Query key information.
- kms:dek:decrypt: Decrypt a data key.
- kms:dek:create: Create a DEK without plaintext.
The authorization will succeed only if you have relevant permissions. Each IAM user only needs to perform the authorization once. Then, perform the following operations:
Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported.
Credential
Select the credential from the drop-down list. If the credential has not been added, click Add Credential, enter a credential name, select an authentication mode, enter the AK/SK pair, and save the changes.
Bucket
Enter the name of the source bucket to be migrated.
NOTE:Symbolic links cannot be used for specifying migration paths which define the migration scope. To migrate a path pointed to by a symbolic link, you need to:- Enter the actual path when specifying the migration path.
- After the migration is complete, manually create a symbolic link to the path at the target.
Endpoint
Enter the endpoint of the region where the source bucket is located.
For example, if Location Type is set to Alibaba Cloud OSS and the source bucket is located in CN East 1 (Hangzhou), enter oss-cn-hangzhou.aliyuncs.com.
Type
Set this parameter based on the source bucket type. Check it in the basic information.
This parameter is available when Location Type is set to Huawei Cloud OBS.
APPID
Enter the APPID of your Tencent Cloud account.
NOTE:You can view the APPID on the account information page of the Tencent Cloud console.
This parameter is available when Tencent Cloud COS is selected for Location Type.
Authentication Type
The following authentication types are supported:
- Connection string
- Storage account name and key
For details about how to obtain the credentials, see How Do I Obtain Credentials for Accessing Microsoft Azure?
This parameter is available when Azure Blob Storage is selected for Location Type.
List Path
Enter the path for storing the list files. These lists must be stored in the same region as the target bucket. You need to write the URLs of files to be migrated and their target object names into the lists. Each line in the list can contain only one URL and one file name.
These parameters are available when HTTP/HTTPS data source is selected for Location Type.
Table 3 Parameters for configuring a migration target Parameter
Description
Remarks
Location Type
Select a Huawei Cloud storage type based on the source storage type.
-
Credential
Select the credential from the drop-down list. If the credential has not been added, click Add Credential, enter a credential name, select an authentication method, enter the AK/SK pair, and click Verify and Save.
These parameters are available when Location Type is set to Huawei Cloud OBS.
Bucket
Select the OBS bucket you are migrating your data to.
NOTE:Only Standard and Infrequent Access buckets are supported.
Type
Set this parameter based on the source bucket type. Check it in the basic information.
Endpoint
Enter the endpoint of the region where the target OBS bucket is located. For example, if the target bucket is located in the CN North-Beijing4 region of Huawei Cloud, enter obs.cn-north-4.myhuaweicloud.com.
NOTE:You can view the endpoint in the OBS bucket overview.
Specify Prefix
Specify a prefix to rename or relocate objects migrated to the target bucket. For example, if you specify the prefix /D, source file /A/B/C.txt will be relocated to /D/A/B/C.txt after being migrated to the target bucket. The prefix can contain up to 1,024 characters. For details, see Adding a Name Prefix or Path Prefix to Migrated Objects.
Enable KMS Encryption
Disabled: The target credential is not encrypted.
These parameters are available when Location Type is set to Huawei Cloud OBS or NAS_SMB.
Enabled: The source credential is encrypted.
Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains:
- kms:cmk:get: Query key information.
- kms:dek:decrypt: Decrypt a data key.
- kms:dek:create: Create a DEK without plaintext.
The authorization will succeed only if you have relevant permissions. Each IAM user only needs to perform the authorization once. Then, perform the following operations:
Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported.
File System Address
Enter the mount address of the target file system. To obtain the mount address, go to the file system list and click the
icon next to the address in the Mount Point column.These parameters are available when Location Type is set to NAS_SMB or NAS_NFS_V3_MOUNT.
Path
Enter the directory for storing migrated files. The format is /<folder-name>.
Username
Enter the username of the account that can access all files in the target file system, for example, administrator.
These parameters are available when Location Type is set to NAS_SMB.
Password
Enter the password corresponding to the username.
Domain on Windows
Enter the corresponding domain name.
NOTE:You only need to enter the content before .com. For example, if the domain is test.com, enter test.
- Configure migration settings based on Table 4.
Table 4 Migration settings Parameter
Sub-parameter
Description
Migration Method
Full migration
This method enables migration of all data in a source bucket or folder.
List migration
This method enables the migration of specific files and folders recorded in predefined lists.
In List Path box, enter the path of the object lists stored in the target bucket. Restrictions on an object list file vary with the target location.
- Target location: Huawei Cloud OBS
- An object list file cannot exceed 30 MB.
- An object list file must be a .txt file, and the Content-Type metadata must be text/plain.
- An object list file must be in UTF-8 without BOM.
- Each line in an object list file can contain only one object name, and the object name must be URL encoded.
- Each line in an object list file cannot exceed 16 KB, or the migration will fail.
- The ContentEncoding metadata of an object list file must be left empty, or the migration will fail.
- An object list file can contain a maximum of 10,000 lines.
- Target location: NAS
- An object list file cannot exceed 30 MB.
- An object list file must be a .txt file.
- An object list file must be in UTF-8 without BOM.
- Each line in an object list file can contain only one object name, and the object name must be URL encoded.
- Each line in an object list file cannot exceed 16 KB, or the migration will fail.
- An object list file can contain a maximum of 10,000 lines.
CAUTION:Prefix migration
This option is only available for cloud storage migration.
If you enter a file name or name prefix in the Prefix box, only the objects that exactly match the specified name or prefix are migrated.
NOTICE:- If the files to be migrated are stored in the root directory of the source bucket, add their name prefixes directly. If the files are stored in a non-root directory, add their path prefixes in the format of <folder-name>/<prefix>.
- Use commas (,) to separate multiple prefixes.
Concurrent Subtasks
-
The default value is 5, but the value can be changed. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, there can be up to 20 subtasks.
Overwrite Existing
Never
Files existing at the target will never be overwritten.
WARNING:- If you choose Never for the initial migration, the attributes of involved parent folders at the source will not be migrated to the target. As a result, the folder attributes may be incomplete at the target. This could impact data integrity, so use the Never option with caution.
- If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.
Always
Files existing at the target will always be overwritten.
If older or different size
- The system replaces existing target files if they are older than or differ in size from their source counterparts. Files with matching modification times and sizes remain unchanged and are skipped during migration.
- The system verifies folders after their content is migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source. NOTE:
The same overwriting policy is applied to empty folders as to files.
If different CRC 64 checksum
If a source object has a CRC64 checksum different from the paired target object, the source object will overwrite the target one. Otherwise, the source object will be skipped during the migration. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked.NOTE:- This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud.
- Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
Consistency Verification
Size and last modified
With this default method, the system checks data consistency by comparing object size and last modification time.
CRC64 checksum
The system checks data consistency by comparing the CRC64 values in the metadata of source and target objects. If a source object or the paired target object does not have a CRC64 checksum, the OMS-calculated CRC64 value is used for verification. CRC64 verification may generate extra public traffic and request costs. For details, see Consistency Verification.NOTE:- This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud.
- Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.
Migrate Metadata
-
Determine whether to migrate metadata.
- If you select this option, object metadata will be migrated.
- If you do not select this option, only the Content-Type and Content-Encoding metadata will be migrated. If the source storage service provider does not provide the metadata (for example, UCloud does not provide Content-Encoding), the metadata will not be migrated.
Clear Cluster
-
Determine whether to clear the migration cluster after the migration is complete.
- If you select this option, a step for clearing the migration cluster will be created in the workflow. You can also choose whether to clear resources used by the cluster, such as NAT gateways, security groups, and VPCEP resources.
- If you do not select this option, a step for clearing the migration cluster will not be created in the workflow, but the migration cluster and its resources will automatically be deleted 30 days after the workflow is created.
- Target location: Huawei Cloud OBS
- (Optional) Configure advanced options based on Table 5.
Table 5 Advanced options Parameter
Description
Target Storage Class
Choose the storage class that your data will be migrated to. For details about storage classes, see How Do I Choose Storage Classes?
Enable KMS Encryption
This parameter is only available for migrations to Huawei Cloud OBS.
- If you do not select this option, whether migrated data will be encrypted in the target buckets depends on the server-side encryption setting of the buckets.
- If you select this option, all migrated objects will be encrypted before they are stored in the target bucket.
NOTE:- Using KMS to encrypt migrated data may slow down the migration speed by about 10%.
- This option is only available when KMS is supported in the region you are migrating to.
Restore Archive Data
- If you do not select this option, the system records archived objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
- If you select this option, the system automatically restores and migrates archived objects in the migration task. If an archived object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
NOTE:The system will restore all archived data found during the listing phase before migration. You need to pay the source cloud platform for the API requests and storage space generated accordingly. To avoid extra restoration fees, you are advised to manually restore the archived data on the source platform before starting the migration.
Filter Source Data
Use exclude patterns, include patterns, and time ranges to filter files to be migrated. For details about filters, see Source Data Filters.
Download Data from CDN
If the default domain name cannot meet your migration requirements, as long as the source cloud service provider supports custom domain names, you can bind a custom domain name to the source bucket, and enable the CDN service on the source platform to reduce data download expenses. Enter a custom domain name in the Domain Name text box and select a transmission protocol. HTTPS is more secure than HTTP and is recommended.
If the migration source is the Alibaba Cloud OSS or Tencent Cloud COS, you also need to select an authentication type and enter an authentication key.
Send SMN Notifications
Determine whether to use SMN to get notifications about migration results.
- If you do not select this option, no SMN messages will be sent after the migration is complete.
- If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.
Limit Traffic
Set the maximum bandwidth for the migration task during different periods.
- If you do not select this option, migration traffic will not be limited.
- If you select this option, you can limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited beyond this period.NOTE:
- The bandwidth limit ranges from 1 MB/s to 1,048,576 MB/s.
- Time periods in different rules cannot overlap.
For example, if there is a rule configured for the period from 8:00 to 12:00, you cannot configure rules for any overlapping periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00.
- Start and end times must be on the same day, with the start time preceding the end time.
For example, the time period from 23:00 to 01:00 of the next day is not allowed.
- A maximum of five rules can be added.
- The time is the local standard time of the region you are migrating to.
Schedule Migration
Schedule the migration to automatically run during a period.
- If you do not select this option, you need to manually start or stop the migration.
- If you select this option, the migration runs during the specified period and stops outside that period.
- If you set Start Time to 08:00 and End Time to 12:00, the migration task runs from 08:00 to 12:00 every day. The migration stops outside that period.
- If you set Start Time to 12:00 and End Time to 08:00, the migration runs from 12:00 of the current day to 08:00 of the next day. The migration stops outside that period.
Step 2: Confirm the Information
- After the migration task is configured, click Next to confirm the information.
- Confirm the settings and click Confirm. The Start Migration Task dialog box is displayed, indicating that the workflow has been created.
- If you want to start the migration immediately, click Confirm.
- If you only want to save the settings and start the migration later, click Cancel. The status of migration tasks becomes Waiting, and the migration has not started. You can click Start in the task list to start the migration.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot