Creating a Migration Task Group

Scenarios

If there is more than 3 TB of data or more than 5 million objects to migrate in a source bucket, you can create a migration task group to migrate the source data quickly. The system will group the source objects into multiple sub-tasks for concurrent migrations.

Prerequisites

• You have registered a HUAWEI ID and enabled Huawei Cloud services.
• You have prepared a URL list and uploaded it to OBS. The URLs in the list can be accessed using HEAD and GET requests.
  

  If the URLs cannot be accessed using HEAD and GET requests, they cannot be migrated by OMS over HTTP or HTTPS. If you still want to use OMS, you are advised to migrate the data from the bucket. Alternatively, contact OBS support.

• You have obtained the AK/SK pairs and required permissions for the destination platform account.
  • The destination platform account needs the permissions required to list buckets, obtain bucket locations, list objects, obtain object metadata, obtain object content, upload objects, list uploaded parts, and restore archive objects. For details about how to obtain these permissions, see Destination Permissions.
• You have created at least one bucket on OBS.
• You have created less than 1,000,000 migration tasks within the past 24 hours.
• You have less than 1,000,000 migration tasks in Waiting state.

Procedure

1. Log in to the Huawei Cloud console.
2. Click in the upper left corner and select the region where the destination bucket is located from the drop-down list.
   Figure 1 Selecting the region where the destination bucket is located
   

3. Click Service List in the upper left corner, and choose Migration > Object Storage Migration Service.
   You can also enter OMS in the search box to quickly find OMS.

   Figure 2 Choosing OMS
   

4. In the navigation pane on the left, choose Migration Task Groups.
5. On the Migration Task Groups page, click Create Migration Task in the upper right corner.
   Figure 3 Creating a migration task group
   

6. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK. The Create Migration Task Group page is displayed.
   Figure 4 Select Source/Destination
   

7. In the Select Source area, select HTTP/HTTPS data sources for Data Source.
8. Set parameters in the Select Destination area and click Next. For details, see Table 1.

   Table 1 Parameters for configuring a destination bucket

   Parameter	Configuration
   Access Key ID	Enter the AK for the Huawei Cloud account to which the destination bucket belongs.
   Secret Access Key	Enter the SK for the Huawei Cloud account to which the destination bucket belongs. Click List Buckets.
   Bucket	Select a destination OBS bucket for storing the migrated data.

9. On the displayed Set Task Parameters page, configure the task parameters.
   Figure 5 Setting task parameters
   

10. Select URL List Folder.
    1. In the URL List Bucket (Huawei Cloud) drop-down list, select the bucket where the list files are stored.
       

       The bucket used for storing the URL lists must be in the same region as the destination bucket, or you cannot select these lists.

    2. Click Select next to Select a URL List File. In the displayed dialog box, select the folder for storing the URL list files and click OK.
       

       Ensure that the URLs in the lists can be accessed using HEAD and GET requests.

11. Determine whether to migrate metadata.
    • If you select Do not migrate, OMS will only migrate the metadata ContentType.
    • If you select Migrate, OMS will migrate supported metadata.
      

      Only Chinese characters, English characters, digits, and hyphens (-) can be migrated. Other characters cannot be migrated.

12. Set Selective Migration.
    • If you do not need to filter source data by time, set this option to No.
    • If you need to filter source data by time, set this option to Yes. Then only source data modified after the specified time will be migrated.

13. Configure parameters in the Destination Configuration area.
    1. Data Encryption If the destination bucket is a parallel file system (PFS), KMS encryption is not supported.
       • If KMS is not selected, all migrated objects will not be encrypted before they are stored into the destination bucket.
       • If KMS is selected, all migrated objects will be encrypted before they are stored into the destination bucket.
         

         • Using KMS to encrypt migrated data may decrease the migration speed by about 10%.
         • Data encryption is only available if KMS encryption is supported in the region you are migrating to.
    2. Specify Prefix
       • If this option is set to No, the names of objects remain unchanged before and after the migration.
       • If this option is set to Yes, you can specify a prefix to rename or redefine the paths of objects migrated to the destination bucket.
         Figure 6 Specifying a prefix
         
         

         To learn how to use Specify Prefix, see Adding a Name Prefix or Path Prefix to Migrated Objects.

    3. Object Storage Class

       Choose the storage class that your data will be migrated to. For details about storage classes, see Introduction to Storage Classes.

14. Click Advanced Settings. Set parameters in Advanced Settings based on Table 2.

    Table 2 Parameters

    Parameter	Description
    Failed Object Record	The default value is Yes. If this option is set to No, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be migrated again. If this option is set to Yes, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, only failed objects need to be migrated again. NOTE: For a migration task, no matter it is contained in a task group or not, the objects that fail to be migrated are recorded in a list named after the task name, and the list is uploaded to the oms/failed_object_lists/ directory of the destination bucket. A maximum of 100,000 failed objects can be recorded in a failed object list file. If more than 100,000 objects fail to be migrated in a migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again.
    SMN Notification	Determine whether to use SMN to get notifications about migration results. The default value is No. If this option is set to No, no SMN message is sent after the migration. If this option is set to Yes, you will receive an SMN message informing you of the migration results.
    Traffic Limiting	Set the maximum bandwidth for the migration task during a specified period of time. If you do not need to limit migration traffic, select No. If you need to limit migration traffic, select Yes.
    Object Overwrite	Overwrite if source newer or different size: If a source object is not as large as or was last modified more recently than its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. Never overwrite: The system always skips source objects and keeps their paired destination objects. Always overwrite: The system always allows source objects to overwrite their paired destination objects.
    Consistency Check	By size and last modification time: This is the default setting. The system checks object consistency with object size and last modification time. No required: This option takes effect for source objects whose sizes cannot be obtained using the content-length field in the standard HTTP protocol. These source objects will overwrite their paired destination objects directly. If the size of a source object can be obtained, its size and last modification time will be checked.
    Priority	If there are a large number of tasks, you can set task priorities to determine the migration sequence.

15. Click Next to enter the confirmation page.
16. Confirm the source information, destination information, task group parameters, and advanced settings, and click Migrate Now. A message is displayed indicating that the task group has been submitted. The task group is displayed in the migration task group list.

Related Operations

You can perform the following operations on a created migration task group.