Help Center/ Object Storage Migration Service/ Best Practices/ Migration from Other Clouds to Huawei Cloud/ Migrating Data from QingCloud QingStor to Huawei Cloud OBS/ Creating a Migration Task
Updated on 2024-10-18 GMT+08:00
View PDF
Share

Creating a Migration Task

This section describes how to create a migration task.

  • By default, if a task stays in waiting state for more than 30 days, OMS automatically puts the task into Failed state and clears the AK/SK pair used by the task. To prevent task failures caused by blocking, do not create tasks too frequently.
  • If an object is larger than 500 GB, migrating it takes a long time. You can submit a service ticket to whitelist the object for migration in segments.
  • During the migration, the system automatically creates a temporary folder named OMS in the target bucket. Do not perform any operations on this folder, including but not limited to modifying, deleting, or adding data in the folder. Otherwise, the migration will be interrupted or fail.

Scenarios

If there is less than 3 TB of data or less than 5 million objects in a source bucket, create a migration task to quickly migrate the source data.

If there is more than 3 TB of data or more than 5 million objects to migrate in a source bucket, create a migration task group to migrate the source data quickly.

Prerequisites

  • You have obtained the AK/SK pairs and required permissions for the source and destination platform accounts.
  • Your source QingCloud account has the permissions to access QingStor.
  • Your Huawei Cloud account has the permissions required to list buckets, obtain bucket locations, list objects, obtain object metadata, obtain object content, upload objects, list uploaded parts, and restore archive objects. For details about how to obtain these permissions, see Destination Permissions.
  • You have created at least one bucket on OBS.
  • You have created less than 1,000,000 migration tasks within the past 24 hours.
  • You have less than 1,000,000 migration tasks in Waiting state.

Procedure

  1. Log in to the Huawei Cloud console.
  2. Click in the upper left corner and select the region where the destination bucket is located from the drop-down list.

    Figure 1 Selecting the region where the destination bucket is located

  3. Click Service List in the upper left corner, and choose Migration > Object Storage Migration Service.

    You can also enter OMS in the search box to quickly find OMS.
    Figure 2 Choosing OMS

  4. On the Migration Tasks page, click Create Migration Task in the upper right corner.

    Figure 3 Creating a migration task

  5. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK. The Create Migration Task page is displayed.

    Figure 4 Select Source/Destination

  6. Set parameters in the Select Source area. For details, see Table 1.

    Table 1 Parameters for configuring a source bucket

    Parameter

    Configuration

    Data Source

    Select QingCloud.

    Access Key ID

    Enter the AK for the QingCloud account that the source bucket belongs to.

    Secret Access Key

    Enter the SK for the QingCloud account that the source bucket belongs to.

    Specify Source Bucket

    Disable Specify Source Bucket (This feature is disabled by default.)

    1. Click List Buckets next to Bucket. All buckets in the account will be listed.
    2. Select the bucket you want to migrate from the drop-down list.
    NOTE:

    If the source account does not have the permissions required to list buckets, enable Specify Source Bucket.

    Enable Specify Source Bucket.

    1. Select the region where the source bucket resides from the Region drop-down list.
    2. In the Bucket text box, enter the name of the bucket or the directory to be migrated. The directory must have the bucket name included.
    3. Click Connect.

  7. Set parameters in the Select Destination area and click Next. For details, see Table 2.

    Table 2 Parameters for configuring a destination bucket

    Parameter

    Configuration

    Access Key ID

    Enter the AK for the Huawei Cloud account to which the destination bucket belongs.

    Secret Access Key

    Enter the SK for the Huawei Cloud account to which the destination bucket belongs.

    Click List Buckets.

    Bucket

    Select a destination OBS bucket for storing the migrated data.

  8. Click Next. The Set Task Parameters page is displayed.

    Figure 5 Set Task Parameters

  9. Select a migration method.

    Migration Method

    Scenario

    Configuration

    File/Folder

    You have obtained the names of the files to be migrated or the folders where the files to be migrated are located.
    • If you select folders, all files in the selected folders will be migrated.
    • If you select files, all selected files will be migrated.

    You can select both folders and files for migration. If you do not select specific files or folders, all objects in the bucket will be migrated by default.

    Object list

    You have obtained the names of all the objects to be migrated, saved the object names in a .txt file, and uploaded the file to an OBS bucket on Huawei Cloud.
    • Object List Bucket (Huawei Cloud): The bucket where the list file is stored must be in the same region as the destination bucket. Otherwise, the list file cannot be selected.
    • Object List: Select the .txt file that contains the names of all objects to be migrated.
    NOTE:
    • An object list file cannot exceed 1,024 MB.
    • The first object in the object list must exist, or the pre-verification will fail.
    • The object list file must be a .txt file, and the Content-Type metadata of the file must be text/plain.
    • The object list file must be in UTF-8 without BOM.
    • Each line in the object list file can contain only one object name, and the object name must be URL encoded.
    • Spaces are not allowed in each line in the object list file. Spaces may cause migration failures because they may be mistakenly identified as object names.
    • The length of each line in the object list file cannot exceed 65,535 characters, or the migration will fail.
    • The Content-Encoding metadata of the object list file must be left empty, or the migration will fail.
    • Archive data can only be accessed and obtained by OMS after it is restored. If the object list file is archive, manually restore it before migration.

    Object name prefix

    You know the naming format of the objects to be migrated, and the objects to be migrated are named with the same prefix.

    After a prefix is added, all objects whose names start with the prefix will be migrated.

    If this parameter is not specified, all objects in the bucket will be migrated.
    • File/Folder
      1. Click Select under the bucket name. The Select File/Folder dialog box is displayed.
      2. Select the folders or files to be migrated. You can select both folders and files for migration. If you do not select specific files or folders, all objects in the bucket will be migrated by default.
      3. Click OK. The selected files or folders are displayed.
        Figure 6 Selected files and folders
    • Object list
      1. In the Object List Bucket (Huawei Cloud) drop-down list, select the bucket where the object list is stored.

        The bucket used for storing object list must be located in the same region as the destination bucket, or you cannot select the object list.

      2. Click Select next to Object List. In the displayed Object List File dialog box, select the .txt file that contains the names of all objects to be migrated, and click OK.
        Figure 7 Selecting an object list file
    • Object name prefix
      1. Enter the names or name prefixes of the objects to be migrated in the text box under Bucket. If no object names or name prefixes are specified, all objects in the source bucket will be migrated.
        Figure 8 Entering the names or name prefixes of objects to be migrated
      2. Click Add. The system will automatically list the matched objects. You can click to delete the objects that do not need to be migrated from the list.

  10. Determine whether to migrate metadata.

    • If you select Do not migrate, OMS will only migrate the metadata ContentType.
    • If you select Migrate, OMS will migrate supported metadata.

      Only Chinese characters, English characters, digits, and hyphens (-) can be migrated. Other characters cannot be migrated.

  11. Set Selective Migration.

    • If you do not need to filter source data by time, set this option to No.
    • If you need to filter source data by time, set this option to Yes. Then only source data modified after the specified time will be migrated.

  12. Configure parameters in the Destination Configuration area.

    1. Data Encryption If the destination bucket is a parallel file system (PFS), KMS encryption is not supported.
      • If KMS is not selected, all migrated objects will not be encrypted before they are stored into the destination bucket.
      • If KMS is selected, all migrated objects will be encrypted before they are stored into the destination bucket.
        • Using KMS to encrypt migrated data may decrease the migration speed by about 10%.
        • Data encryption is only available if KMS encryption is supported in the region you are migrating to.
    2. Specify Prefix
      • If this option is set to No, the names of objects remain unchanged before and after the migration.
      • If this option is set to Yes, you can specify a prefix to rename or redefine the paths of objects migrated to the destination bucket.
        Figure 9 Specifying a prefix

        To learn how to use Specify Prefix, see Adding a Name Prefix or Path Prefix to Migrated Objects.

    3. Object Storage Class

      Choose the storage class that your data will be migrated to. For details about storage classes, see Introduction to Storage Classes.

  13. Click Advanced Settings. Set parameters in the Advanced Settings area based on Table 3.

    Table 3 Parameters

    Parameter

    Configuration

    Failed Object Record

    The default value is Yes.

    • If this option is set to No, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be migrated again.
    • If this option is set to Yes, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, only failed objects need to be migrated again.
      NOTE:
      • For a migration task, no matter it is contained in a task group or not, the objects that fail to be migrated are recorded in a list named after the task name, and the list is uploaded to the oms/failed_object_lists/ directory of the destination bucket.
      • A maximum of 100,000 failed objects can be recorded in the failed object list.

        If more than 100,000 objects fail to be migrated in a migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again.

    Archive Data Restoration

    The default value is No.

    • If this option is set to No, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects.
    • If this option is set to Yes, the system automatically restores and migrates archive objects.

    SMN Notification

    Determine whether to use SMN to get notifications about migration results. The default value is No.

    • If this option is set to No, no SMN message is sent after the migration.
    • If this option is set to Yes, you will receive an SMN message informing you of the migration result.

    Traffic Limiting

    Set the maximum bandwidth for the migration task during a specified period of time.

    • If you do not need to limit migration traffic, select No.
    • If you need to limit migration traffic, select Yes.

    Object Overwrite
    • Overwrite if source newer or different size: If a source object is not as large as or was last modified more recently than its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped.
    • Never overwrite: The system always skips source objects and keeps their paired destination objects.
    • Overwrite if different CRC64 checksum: If a source object has a CRC64 checksum different from the paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked.
      NOTE:
      • This option is only available for migration on Huawei Cloud or from Alibaba Cloud or Tencent Cloud.
      • This option is applied only when the source object has the same encryption status as the destination one.
    • Always overwrite: The system always allows source objects to overwrite their paired destination objects.

    Consistency Check
    • By size and last modification time: This is the default setting. The system checks object consistency with object size and last modification time.
    • By CRC64 checksum: The system checks object consistency with CRC64 checksums. If a source object and the paired destination object have CRC64 checksums, the checksums are checked. Otherwise, their sizes and last modification times are checked.

    Priority

    If there are a large number of tasks, you can set task priorities to determine the migration sequence.

  14. Click Next to enter the confirmation page.
  15. Confirm the source information, destination information, task parameters, and advanced settings, and click Migrate Now. A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.

    After the task is created, a task ID is generated for quick identification. It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

Related Operations

You can perform the following operations on a created migration task.

Operation

Description

View a migration task.

For details, see Viewing a Migration Task.

Manage a migration task.

For details, see Managing a Migration Task.