Creating a Migration Task Group_Object Storage Migration Service

Scenario

If there is more than 3 TB of data or more than 5 million objects to migrate in a source bucket, you can create a migration task group to migrate the source data quickly. The system will group the source objects into multiple sub-tasks for concurrent migrations.

By default, if a task stays in waiting state for more than 30 days, OMS automatically puts the task into Failed state and clears the AK/SK pair used by the task. To prevent task failures caused by blocking, do not create tasks too frequently.
If an object is larger than 500 GB, migrating it takes a long time. You can submit a service ticket to whitelist the object for migration in segments.

Background

OMS supports the following migration sources:

Huawei Cloud
AWS
Alibaba Cloud
Microsoft Azure
Tencent Cloud
Google Cloud
Qiniu Cloud
QingCloud
Baidu Cloud
Kingsoft Cloud
UCloud
HTTP/HTTPS data sources

This section uses Huawei Cloud as the migration source. For tutorials for other cloud platforms, see Migration from Other Clouds to Huawei Cloud.

OMS makes API calls to the object storage services on source platforms during the migration. These API calls may generate fees on those platforms and you will be responsible for those fees based on those source platform billing structures.

Prerequisites

You have registered a HUAWEI ID and enabled Huawei Cloud services.
You have obtained the AK/SK pairs and required permissions for the source and destination platform accounts.
- The source platform account needs the permissions for:
  - Listing all buckets
  - Obtaining bucket locations
  - Listing objects
  - Obtaining object metadata
  - Obtaining object content
  To learn how to obtain these permissions, see Source Permissions.
- The destination platform account needs the permissions for:
  - Listing all buckets
  - Obtaining bucket locations
  - Listing objects
  - Obtaining object metadata
  - Obtaining object content
  - Listing uploaded parts
  - Restoring archive objects
  To learn how to obtain these permissions, see Destination Permissions.
You have created at least one bucket on OBS.
You have created less than 300 migration task groups within the past 24 hours.
You have created less than 1,000,000 migration tasks within the past 24 hours.
You have less than 1,000,000 migration tasks in Waiting state.

Procedure

Log in to the management console.
Click in the upper left corner and select the desired region.
Click Service List. Under Migration, click Object Storage Migration Service.

The Object Storage Migration Service page is displayed.
In the navigation pane on the left, choose Migration Task Groups.
Click Create Migration Task Group in the upper right corner.

Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK.

The page for creating a migration task group is displayed.

For step Select Source/Destination, perform the following operations:

Set the source parameters by referring to Table 1.

**Table 1** Source parameters
Parameter	Description
Data Source	Select the source cloud service provider.
(Optional) appID	Enter the identifier of the account for logging in to the Tencent Cloud console. This parameter is only available if the source cloud service provider is Tencent Cloud.
Access Key ID	Enter the AK for accessing the source platform. It can contain a maximum of 100 characters.
Secret Access Key	Enter the SK for accessing the source platform. The SK matches the AK, and they group into a pair. It can contain a maximum of 100 characters.
Specify Source Bucket	Enable this option if your source account does not have the permissions required to list buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects. Then specify the bucket or path to be migrated. For details, see Table 2.

**Table 2** Description of **Specify Source Bucket**
If You...	Then...
Enable Specify Source Bucket	Select the region where the source bucket is located from the Region drop-down list. In the Bucket text box, enter the bucket or the path to be migrated. The path must have the bucket name included. Click Connect. NOTE: OMS will check whether the source platform can be connected using the information you entered. If the source platform can be connected, the system will prompt you.
Disable Specify Source Bucket (This feature is disabled by default.)	Click List Buckets. OMS will check whether the source platform can be connected using the information you entered. After the source platform is connected, all buckets in your account will be listed in the Bucket drop-down list on the premises that your account has the permissions needed to list buckets. In the Bucket drop-down list, select the bucket you want to migrate.

Set the destination parameters listed in Table 3.

**Table 3** Destination parameters
Parameter	Description
Access Key ID	Enter the AK for accessing Huawei Cloud. It can contain a maximum of 100 characters.
Secret Access Key	Enter the SK for accessing Huawei Cloud. The SK matches the AK, and they group into a pair. It can contain a maximum of 100 characters. Then click List Buckets.
Bucket	Select a destination OBS bucket for storing the migrated data.

Click Next. The Set Task Group Parameters page is displayed.

Set parameters in the Source Configuration area.

Select a migration method.

Migration Method	Description
Object name prefix	Enter the names or prefixes of objects to be migrated and click Add. The system will automatically list the matched objects. You can click to delete a prefix, and then the system will not migrate the objects matched with the deleted prefix. NOTE: If the objects you want to migrate are stored in the root directory of the source bucket, add their prefixes directly. If these objects are stored in none-root directories, add their directories and prefixes in the format of Bucket name/Folder/Prefix. If you do not add any object names or prefixes, all objects in the source bucket will be migrated. The objects to be migrated are recorded in .txt files, and each migration task has a list file. The storage path of the files is oms/taskgroup/ in the destination bucket. These list files will be retained after the migration, and you can manually delete them.
Object list folder	Generally, this method is used to migrate incremental objects. You need to list the names of source objects to be migrated in TXT files and store the files in an OBS bucket on Huawei Cloud. OMS will migrate all objects specified in the files. Select the bucket where the object lists are stored. NOTE: The listed buckets are obtained using the destination AK and SK you entered in 6.b. The bucket used for storing object lists must be in the same region as the destination bucket, or you cannot select these object lists. Click Select next to Object List Folder, select the object list files, and click OK. NOTE: An object list file cannot exceed 1,024 MB. The total number of files in the object list folder cannot exceed 2,000. An object list file must be a .txt file, and the Content-Type metadata of the file must be text/plain. An object list file must be in UTF-8 without BOM. Each line in the object list file can contain only one object name, and the object name must be URL encoded if it contains special characters or spaces. Spaces are not allowed in each line in an object list file. Spaces may cause migration failures because they may be mistakenly identified as object names. The length of each line in an object list file cannot exceed 65,535 characters, or the migration will fail. The Content-Encoding metadata of each object list file must be left empty, or the migration will fail. Archive data can only be accessed and obtained by OMS after it is restored. If the object list files are archive, manually restore them before migration.
URL list folder	Generally, this method is used to migrate HTTP/HTTPS data. You need to write the URLs of the files to be migrated and their destination objects names into one or more .txt files, and store the files in an OBS bucket on Huawei Cloud. You can store up to 2,000 list files in a fixed folder in the OBS bucket. Each list file cannot exceed 1 GB. OMS migrates all objects specified in the URL list files. Select the bucket where the list files are stored. NOTE: The listed buckets are obtained using the destination AK and SK you entered in 6.b. The bucket used for storing the list files must be located in the same region as the destination bucket, or you cannot select these files. For security purposes, if the migration source supports both HTTP and HTTPS, you are advised to use HTTPS during the migration. If HTTP is used, data may be eavesdropped on, tampered with, or vulnerable to man-in-the-middle or replay attacks. If the data to be migrated is not public and you do not want to make the data public, you are advised to use other migration methods. URL redirection is not supported. Click Select next to URL List Folder, select the URL list files, and click OK. NOTE: A URL list file cannot exceed 1,024 MB. The total number of files in the URL list folder cannot exceed 2,000. A URL list file must be a .txt file, and the Content-Type metadata of the file must be text/plain. A URL list file must be in UTF-8 without BOM. Each line in a URL list file can contain only one URL and one destination object name. The length of each line in a URL list file cannot exceed 65,535 characters, or the migration will fail. The Content-Encoding metadata of each URL list file must be left empty, or the migration will fail. In the URL list file, you must use a tab character (\t) to separate the URL and destination object name in each line. Do not use spaces to separate them. The format is [URL][Tab character][Destination object name]. Only the Chinese and special characters in the source and destination object names must be URL encoded. Please note that: The URL encoding starts from the second character after the domain name in a line. Do not encode the protocol header, domain name, or slash before or after the domain name. Otherwise, the format verification will fail. In each line, use a tab character (Tab key on the keyboard) to separate the URL and the destination object name. Do not use spaces. For example: http://xxx.xxx.xxx.xxx.com/doc/thefile2.txt doc/thefile2.txt http://xxx.xxx.xxx.xxx.com/the%20file.txt the%20file.txt http://xxx.xxx.xxx.xxx.com/the%20file2.txt the+file2.txt http://xxx.xxx.xxx.xxx.com/doc/thefile.txt doc/thefile.txt In the preceding examples, after the files represented by the URLs are copied to the destination bucket, the objects are named doc/the file2.txt, the file.txt, the file2.txt, and doc/thefile.txt. Spaces are not allowed in each line in the URL list file. Spaces may cause migration failures because they may be mistakenly identified as object names.

Migration Method

Description

Object name prefix

Enter the names or prefixes of objects to be migrated and click Add. The system will automatically list the matched objects. You can click to delete a prefix, and then the system will not migrate the objects matched with the deleted prefix.

NOTE:

If the objects you want to migrate are stored in the root directory of the source bucket, add their prefixes directly. If these objects are stored in none-root directories, add their directories and prefixes in the format of Bucket name/Folder/Prefix.
If you do not add any object names or prefixes, all objects in the source bucket will be migrated.
The objects to be migrated are recorded in .txt files, and each migration task has a list file. The storage path of the files is oms/taskgroup/ in the destination bucket. These list files will be retained after the migration, and you can manually delete them.

Object list folder

Generally, this method is used to migrate incremental objects. You need to list the names of source objects to be migrated in TXT files and store the files in an OBS bucket on Huawei Cloud. OMS will migrate all objects specified in the files.

Select the bucket where the object lists are stored.
NOTE:

The listed buckets are obtained using the destination AK and SK you entered in 6.b.

The bucket used for storing object lists must be in the same region as the destination bucket, or you cannot select these object lists.
Click Select next to Object List Folder, select the object list files, and click OK.

NOTE:

An object list file cannot exceed 1,024 MB.
The total number of files in the object list folder cannot exceed 2,000.
An object list file must be a .txt file, and the Content-Type metadata of the file must be text/plain.
An object list file must be in UTF-8 without BOM.
Each line in the object list file can contain only one object name, and the object name must be URL encoded if it contains special characters or spaces.
Spaces are not allowed in each line in an object list file. Spaces may cause migration failures because they may be mistakenly identified as object names.
The length of each line in an object list file cannot exceed 65,535 characters, or the migration will fail.
The Content-Encoding metadata of each object list file must be left empty, or the migration will fail.
Archive data can only be accessed and obtained by OMS after it is restored. If the object list files are archive, manually restore them before migration.

URL list folder

Generally, this method is used to migrate HTTP/HTTPS data. You need to write the URLs of the files to be migrated and their destination objects names into one or more .txt files, and store the files in an OBS bucket on Huawei Cloud. You can store up to 2,000 list files in a fixed folder in the OBS bucket. Each list file cannot exceed 1 GB. OMS migrates all objects specified in the URL list files.

Select the bucket where the list files are stored.
NOTE:
- The listed buckets are obtained using the destination AK and SK you entered in 6.b.
- The bucket used for storing the list files must be located in the same region as the destination bucket, or you cannot select these files.
- For security purposes, if the migration source supports both HTTP and HTTPS, you are advised to use HTTPS during the migration. If HTTP is used, data may be eavesdropped on, tampered with, or vulnerable to man-in-the-middle or replay attacks.
- If the data to be migrated is not public and you do not want to make the data public, you are advised to use other migration methods.
- URL redirection is not supported.
Click Select next to URL List Folder, select the URL list files, and click OK.
NOTE:
- A URL list file cannot exceed 1,024 MB.
- The total number of files in the URL list folder cannot exceed 2,000.
- A URL list file must be a .txt file, and the Content-Type metadata of the file must be text/plain.
- A URL list file must be in UTF-8 without BOM.
- Each line in a URL list file can contain only one URL and one destination object name.
- The length of each line in a URL list file cannot exceed 65,535 characters, or the migration will fail.
- The Content-Encoding metadata of each URL list file must be left empty, or the migration will fail.
- In the URL list file, you must use a tab character (\t) to separate the URL and destination object name in each line. Do not use spaces to separate them. The format is [URL][Tab character][Destination object name]. Only the Chinese and special characters in the source and destination object names must be URL encoded.
  Please note that:
  
  The URL encoding starts from the second character after the domain name in a line. Do not encode the protocol header, domain name, or slash before or after the domain name. Otherwise, the format verification will fail.
  
  In each line, use a tab character (Tab key on the keyboard) to separate the URL and the destination object name. Do not use spaces.
  For example:
  http://xxx.xxx.xxx.xxx.com/doc/thefile2.txt doc/thefile2.txt http://xxx.xxx.xxx.xxx.com/the%20file.txt the%20file.txt http://xxx.xxx.xxx.xxx.com/the%20file2.txt the+file2.txt http://xxx.xxx.xxx.xxx.com/doc/thefile.txt doc/thefile.txt
  In the preceding examples, after the files represented by the URLs are copied to the destination bucket, the objects are named doc/the file2.txt, the file.txt, the file2.txt, and doc/thefile.txt.
- Spaces are not allowed in each line in the URL list file. Spaces may cause migration failures because they may be mistakenly identified as object names.

Determine whether to migrate metadata.
- If you select Do not migrate, OMS will only migrate the metadata Content-Type.
- If you select Migrate, OMS will migrate supported metadata.
  
  Only English characters, numbers, and hyphens (-) in metadata can be migrated. Chinese characters and symbols cannot be migrated, regardless of whether they are encoded.
Set Incremental Migration.
- If you do not need to filter source data by time, set this option to No.
- If you need to filter source data by time, set this option to Yes. Then only source data modified after the specified time will be migrated.

Configure parameters in the Destination Configuration area.
1. Data Encryption If the destination bucket is a parallel file system (PFS), KMS encryption is not supported.
  - If KMS is not selected, all migrated objects will not be encrypted before they are stored into the destination bucket.
  - If KMS is selected, all migrated objects will be encrypted before they are stored into the destination bucket.
    Using KMS to encrypt migrated data may decrease the migration speed by about 10%.
    
    Data encryption is only available if KMS encryption is supported in the region you are migrating to.
2. Specify Prefix
  - If this option is set to No, the names of objects remain unchanged before and after the migration.
  - If this option is set to Yes, you can specify a prefix, and the configured prefix will be added to the names of objects migrated to the destination bucket.
    Figure 1 Specifying a prefix
    
    To learn how to use Specify Prefix, see Adding a Name Prefix or Path Prefix to Migrated Objects.
3. Object Storage Class
  Choose the storage class that your data will be migrated to. For details about storage classes, see Introduction to Storage Classes.

(Optional) Click Show next to Advanced Settings to expand the advanced options. Configure advanced options by referring to Table 4.

**Table 4** **Advanced Settings** parameters
Parameter	Configuration
Failed Object Record	Decide whether to record the objects that fail to be migrated. If this option is set to No, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be migrated again. If this option is set to Yes, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, only failed objects need to be migrated again. NOTE: For a migration task, no matter it is contained in a task group or not, the objects that fail to be migrated are recorded in a list named after the task name, and the list is uploaded to the oms/failed_object_lists/ directory of the destination bucket. A maximum of 100,000 failed objects can be recorded in the failed object list. If more than 100,000 objects fail to be migrated in the migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again. This file contains information about the failed migration task, including the failure cause and time (in local UTC), URL-encoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs. This file is retained in the destination bucket even after the objects in the file are successfully re-transferred. If you do not need the file, delete it manually.
Archive Data Restoration	Archive data must be restored before being migrated. You can select this option if the source cloud service provider supports automatic restoration of archive data. If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. CAUTION: Currently, the following cloud platforms can automatically restore archive objects: Huawei Cloud, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud. The system will restore archive data before migrating it, and you will pay to the source cloud service provider for the API requests and storage space generated accordingly. The copies generated for restoring archive data will be stored for three days by default. If there is more than 3 TB of data or more than 3 million objects in a source bucket, create a migration task group to migrate data in case the migration fails because the restored data is archived again. Pausing or restarting a migration task will result in restoration of all archive data again, no matter it has been restored or not. After data is migrated from the source platform to Huawei Cloud, the storage class of the migrated data is the same as that of the destination bucket. If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated, skips them, and continues to migrate other objects in the migration task.
SMN Notification	Decide whether to use SMN to get notifications about migration results. Set to No No SMN message is sent after the migration. Set to Yes You will receive an SMN message informing you of the migration results. Perform the following steps to set SMN parameters: Click Add next to SMN Topic. The Select SMN Topic dialog box is displayed. Select a topic for receiving SMN messages and click OK. If OMS is not authorized to publish messages to the selected SMN topic, a Warning dialog box will display after you click OK. If you need the authorization, click OK to authorize OMS. Otherwise, click Cancel to return to the Select SMN Topic page and select another SMN topic. Configure Trigger Condition. You can select one of the following options or both: Migration succeeded An SMN message will be sent after the migration task succeeds. Migration failed An SMN message will be sent after the migration task fails. NOTE: If you experience trouble in using SMN, see What What Can I Do If SMN Is Not Working Properly?
Traffic Limiting	Set the maximum bandwidth for the migration task during a specified period. If you do not need to limit migration traffic, select No. If you need to limit migration traffic, select Yes. Perform the following steps to set traffic limits: Click Add Rule. A traffic limiting rule is displayed. Specify Start Time, End Time, and Maximum Bandwidth. The maximum bandwidth ranges from 1 MB/s to 200 MB/s. NOTICE: For example, if you set Start Time to 08:00, End Time to 12:00, and Maximum Bandwidth to 20 MB/s, the maximum migration rate is limited to 20 MB/s when the migration task is running in the period from 08:00 to 12:00. The migration speed is not limited beyond this period. The time is the local standard time of the region you selected. Repeat 1 and 2 to add another traffic limiting rule. You can click in the Operation column to delete a rule. NOTE: A maximum of five rules can be added for a migration task. You can set traffic limiting rules in the task details even after the task is created.
Object Overwrite	Overwrite if source newer or different size: If a source object is not as large as or was last modified more recently than its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. Never overwrite: The system always skips source objects and keeps their paired destination objects. Overwrite if different CRC64 checksum: If a source object has a CRC64 checksum different from the paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked. NOTE: This option is only available for migration on Huawei Cloud or from Alibaba Cloud or Tencent Cloud. This option is applied only when the source object has the same encryption status as the destination one. Always overwrite: The system always allows source objects to overwrite their paired destination objects.
Consistency Check	By size and last modification time: This is the default setting. The system checks object consistency with object size and last modification time. By CRC64 checksum: The system checks object consistency with CRC64 checksums. If a source object and the paired destination object have CRC64 checksums, the checksums are checked. Otherwise, their sizes and last modification times are checked.
Priority	If there are a large number of tasks, you can set task priorities to determine the migration sequence.
Description	In the Description area, enter a maximum of 255 characters to briefly describe the migration task.

Click Next to enter the confirmation page.
Confirm the migration settings and start the migration.
1. Check whether the settings of migration parameters are correct.
  If the settings are incorrect, click Previous to return to the previous page for modification.
2. Click Migrate Now. A message is displayed, indicating that the migration task is created successfully.
  The task group is displayed in the migration task group list.

Creating a Migration Task Group

Scenario

Background

Prerequisites

Procedure

Feedback

Was this page helpful?