Creating a Migration Task

Scenario

If there is less than 3 TB of data or less than 5 million objects in a source bucket, you can create a migration task to quickly migrate the source data.

If there is more than 3 TB of data or more than 5 million objects to migrate in a source bucket, you can create a migration task group to migrate the source data quickly.

Prerequisites

• You have registered a HUAWEI ID and enabled Huawei Cloud services.
• You have obtained the AK/SK pairs and required permissions for the source and destination platform accounts.
  • The source platform account needs the KS3ReadOnlyAccess permission.
  • The destination platform account needs the permissions required to list buckets, obtain bucket locations, list objects, obtain object metadata, obtain object content, upload objects, list uploaded parts, and restore archive objects. For details about how to obtain these permissions, see Destination Permissions.
• You have created a bucket in OBS.
• You have created less than 1,000,000 migration tasks within the past 24 hours.
• You have less than 1,000,000 migration tasks in Waiting state.

Procedure

1. Log in to the Huawei Cloud console.
2. Click in the upper left corner and select the region where the destination bucket is located from the drop-down list.
   Figure 1 Selecting the region where the destination bucket is located
   

3. Click Service List in the upper left corner, and choose Migration > Object Storage Migration Service.
   You can also enter OMS in the search box to quickly find OMS.

   Figure 2 Choosing OMS
   

4. On the Migration Tasks page, click Create Migration Task in the upper right corner.
   Figure 3 Creating a migration task
   

5. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK. The Create Migration Task page is displayed.
   Figure 4 Select Source/Destination
   

6. Set parameters in the Select Source area. For details, see Table 1.

   Table 1 Parameters required for configuring a source bucket

   Parameter	Configuration
   Data Source	Select Kingsoft Cloud.
   Access Key ID	Enter the AK of the Kingsoft Cloud account to which the source bucket belongs.
   Secret Access Key	Enter the SK of the Kingsoft Cloud account to which the source bucket belongs.
   Specify Source Bucket	Disable Specify Source Bucket (This feature is disabled by default.) Select the bucket where the source data is stored from the Bucket drop-down list. OMS will check whether the source platform can be connected using the information you provided. Once the source platform is connected, all buckets in your account will be listed in the Bucket drop-down list, provided your account has the necessary permissions to list buckets. NOTE: If the source platform account does not have the permissions required to list buckets, you need to enable Specify Source Bucket.
   	Enable Specify Source Bucket. Select the region where the source bucket resides from the Region drop-down list. In the Bucket text box, enter the name of the bucket or the directory to be migrated. The directory must have the bucket name included. Click Connect.

7. Set parameters in the Select Destination area and click Next. For details, see Table 2.

   Table 2 Parameters for configuring a destination bucket

   Parameter	Configuration
   Access Key ID	Enter the AK for the Huawei Cloud account to which the destination bucket belongs.
   Secret Access Key	Enter the SK for the Huawei Cloud account to which the destination bucket belongs.
   Bucket	Select a destination OBS bucket for storing the migrated data.

8. Click Next. The Set Task Parameters page is displayed.
   Figure 5 Set Task Parameters
   

9. Select a migration method.

   Migration Method	Scenario	Description
   File/Folder	You have obtained the names of the files to be migrated or the folders where the files to be migrated are located.	If you select folders, all files in the selected folders will be migrated. If you select files, all selected files will be migrated. You can select both folders and files for migration. If you do not select specific files or folders, all objects in the bucket will be migrated by default.
   Object list	You have obtained the names of all the objects to be migrated, saved the object names in a .txt file, and uploaded the file to an OBS bucket on Huawei Cloud.	Object List Bucket (Huawei Cloud): The bucket where the list file is stored must be in the same region as the destination bucket. Otherwise, the list file cannot be selected. Object List: Select the .txt file that contains the names of all objects to be migrated. NOTE: Each object list file cannot exceed 1,024 MB. The first object in the object list must exist, or the pre-verification will fail. The object list file must be a .txt file, and the ContentType metadata of the file must be text/plain. The object list file must be in UTF-8 without BOM. Each line in the object list file can contain only one object name, and the object name must be URL encoded. Spaces are not allowed in each line in the object list file. Spaces may cause migration failures because they may be mistakenly identified as object names. The length of each line in the object list file cannot exceed 65,535 characters, or the migration will fail. ContentEncoding cannot be set in the metadata of the object list file, or the migration will fail. Archive data can only be accessed and obtained by OMS after it is restored. If the object list files are archive, manually restore them before migration.
   Object name prefix	You know the naming format of the objects to be migrated, and the objects to be migrated are named with the same prefix.	After a prefix is added, all objects whose names start with the prefix will be migrated. If this parameter is not specified, all objects in the bucket will be migrated.

   • File/Folder
     1. Click Select under the bucket name. The Select File/Folder dialog box is displayed.
     2. Select the folders or files to be migrated. You can select both folders and files for migration. If you do not select specific files or folders, all objects in the bucket will be migrated by default.
     3. Click OK. The selected files or folders are displayed.
        Figure 6 Selected files and folders
        
   • Object list
     1. Under Object List Bucket (Huawei Cloud), select the bucket where the object list is stored.
        

        The bucket used for storing object list must be located in the same region as the destination bucket, or you cannot select the object list.

     2. Click Select next to Object List. In the displayed Object List File dialog box, select the .txt file that contains the names of all objects to be migrated, and click OK.
        Figure 7 Selecting an object list file
        
   • Object name prefix
     1. Enter the names or name prefixes of the objects to be migrated in the text box under Bucket. If no object names or name prefixes are specified, all objects in the source bucket will be migrated.
        Figure 8 Entering a prefix
        
     2. Click Add. The system will automatically list the matched objects. You can click to delete the objects that do not need to be migrated from the list.

10. (Optional) Set Use Source Custom Domain Name based on Table 3.

    

    Table 3 Parameters

    Parameter	Configuration
    Use Source Custom Domain Name	It is disabled by default. If the default domain name cannot meet your migration requirements, then as long as the source cloud service provider supports custom domain names, you can bind a custom domain name to the source bucket, and enable the CDN service on the source platform to reduce data download fees. NOTE: If anti-leeching is used, you are advised to disable it or contact Huawei Cloud technical support. Select a transmission protocol for Transfer Protocol. NOTE: HTTPS is more secure than HTTP and is recommended. Enter a custom domain name of the bucket in the Custom Bucket Domain Name box.
    Check	Click Check to check whether the configuration is correct. If the check fails, an error message is displayed in the upper right corner. If the check passes, complete other migration configurations.

11. Configure parameters in the Destination Configuration area.
    1. Specify Prefix
       • If you choose to disable this feature, the names of objects remain unchanged before and after the migration.
       • If choose to enable this feature, you can specify a prefix to rename or redefine the paths for objects migrated to the destination bucket.
         Figure 9 Entering a prefix
         
         

         To learn how to use Specify Prefix, see Adding a Name Prefix or Path Prefix to Migrated Objects.

    2. Object Storage Class

       Choose the storage class that your data will be migrated to. For details about storage classes, see Storage Classes.

12. Click before Advanced Settings. Set parameters in the Advanced Settings area based on Table 4.

    Table 4 Advanced settings

    Parameter	Configuration
    Object Metadata	Determine whether to migrate metadata. If you select Do not migrate, OMS will only migrate the Content-Type and Content-Encoding metadata. If you select Migrate, OMS will migrate supported metadata. NOTICE: Only Chinese characters, English characters, digits, and hyphens (-) can be migrated. Other characters cannot be migrated. Even if you choose not to migrate metadata, the Content-Type and Content-Encoding metadata will still be migrated to ensure that the migration runs properly.
    Selective Migration	Determine whether to filter source data to be migrated by time. If you do not need to filter source data by time, disable this option. If you need to filter source data by time, enable this option and specify a date and time. Then only source data modified after the specified date and time will be migrated.
    Server-Side Encryption	Determine whether to enable server-side encryption. If you choose to disable this option, objects will not be encrypted before they are stored to the destination bucket. If you choose to enable this option, objects will be encrypted before they are stored to the destination bucket. NOTE: SSE-KMS is used by default. Other encryption methods or custom encryption is not supported. If the destination is a parallel file system (PFS), KMS encryption is not supported. Enabling KMS-based server-side encryption may slow down the migration speed by about 10%. KMS encryption is only available if KMS is supported in the region you are migrating to.
    Failed Object Record	Determine whether to record the objects that fail to be migrated. If you choose to disable this option, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be migrated again. If you choose to enable this option, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, you can choose to only migrate failed objects again. NOTE: For a migration task, no matter it is contained in a task group or not, the objects that fail to be migrated are recorded in a list named after the task name, and the list is uploaded to the oms/failed_object_lists/ directory of the destination bucket. A maximum of 100,000 failed objects can be recorded in the failed object list. If more than 100,000 objects fail to be migrated in the migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again. This file contains information about the failed migration task, including the failure cause and time (in local UTC), URL-encoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs. This file is retained in the destination bucket even after the objects in the file are successfully re-transferred. If you do not need the file, delete it manually.
    Archive Data Restoration	Archived data must be restored before it can be migrated. You can choose to enable this option if your source cloud service provider supports automatic restoration of archived data. If you enable this option, the system automatically restores and migrates archived objects in the migration task. If an archive object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. NOTICE: The following cloud service providers can automatically restore archived objects: Huawei Cloud, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud. It takes at least a minute to restore an archived object. The larger the archived object is, the longer it takes to restore it, but if an archived object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see the documentation of your source cloud service provider. Two types of fees are generated during archived data restoration: fees for API calls and fees for archived object restoration. These fees are billed by the source cloud service provider. After the restoration is complete, archived objects will be in restored state for a certain period of time. During this period, objects can be downloaded or read. After this period expires, archived objects need to be restored again for migration. For details about how long they stay in the restored state, see the documentation of the cloud service provider. The copies generated for restoring archived objects will be stored for three days by default. If there is more than 3 TB of data or more than 3 million objects in a source bucket, create a migration task group to migrate data in case the migration fails because the restored data is archived again. Pausing or restarting a migration task will result in restoration of all archive data again, no matter it has been restored or not. After objects are migrated from the source platform to Huawei Cloud, they are in the same storage class as the destination bucket. If you do not enable this option, the system directly records archived objects in the list of objects that failed to be migrated, skips them, and continues to migrate other objects in the migration task.
    SMN Notification	Decide whether to use SMN to get notifications about migration results. If you do not enable this option, no SMN message will be sent after the migration is complete. If you enable this option, you will receive an SMN message informing you of the migration results. Perform the following steps: Click Add next to the SMN Topic box. The Select SMN Topic dialog box is displayed. Select a topic for publishing SMN messages and click OK. If OMS is not authorized to publish messages to the selected SMN topic, a Warning dialog box will appear after you click OK. If you need to complete the authorization, click OK to authorize OMS. Otherwise, click Cancel to return to the Select SMN Topic page and select another SMN topic. Click Add next to the SMN Template box. The Select SMN Template dialog box is displayed. If this parameter is specified, SMN messages will be sent using the specified template. The template must use the default protocol. Select a template for publishing SMN messages and click OK. Configure Trigger Condition. You can select one of the following options or both: Migration succeeded: An SMN message will be sent if the migration task succeeds. Migration failed: An SMN message will be sent if the migration task fails. NOTE: If you experience trouble in using SMN, see What Do I Do If SMN Does Not Work Properly?
    Traffic Limiting	Set the maximum bandwidth for the migration task during different periods. If you do not need to limit the migration traffic, disable this option. If you need to limit the migration traffic, enable this option. Perform the following steps: Specify Start Time, End Time, and Maximum Bandwidth. The maximum bandwidth ranges from 1 MB/s to 200 MB/s. NOTICE: For example, if you set Start Time to 08:00, End Time to 12:00, and Maximum Bandwidth to 20 MB/s, the maximum migration speed is limited to 20 MB/s when the migration task runs in the period from 08:00 to 12:00. The migration speed is not limited beyond this period. The time is the local standard time of the region you are migrating to. Click Add Rule. An empty traffic limiting rule is displayed. Repeat 1 and 2 to add another traffic limiting rule. You can click Delete in the Operation column to delete a rule. NOTE: A maximum of five rules can be added for a migration task. You can set traffic limiting rules in the task details even after the task is created.
    Object Overwrite	Overwrite if source newer or different size: If a source object is not as large as or was last modified more recently than its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. Never overwrite: The system will always skip source objects and keeps their paired destination objects. Overwrite if different CRC64 checksum: If a source object has a CRC64 checksum different from the paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked. NOTE: This option is only available for migration on Huawei Cloud or from Alibaba Cloud or Tencent Cloud. This option is applied only when the source object has a same encryption attribute as the destination one. Always overwrite: The system always allows source objects to overwrite their paired destination objects.
    Consistency Check	By size and last modification time: With this default setting, the system will check object consistency with object size and last modification time. By CRC64 checksum: The system will check object consistency with CRC64 checksums. If a source object and the paired destination object have CRC64 checksums, the checksums are checked. Otherwise, their sizes and last modification times are checked.
    Priority	If there are a large number of tasks, you can set task priorities to control which tasks take precedence over other tasks.
    Description	In the Description area, enter a maximum of 255 characters to briefly describe the migration task.

13. Click Next to enter the confirmation page.
14. Confirm the source information, destination information, task parameters, and advanced settings, and click Next. A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.
    

    After the task is created, a task ID is generated for quick identification. It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

Related Operations

You can perform the following operations on a created migration task.