Object Storage Migration Task (Shared Cluster)

What Is Object Storage Migration Task (Shared Cluster)?

This solution is developed based on public computing resources. It helps you migrate object storage data from other cloud vendors to Huawei Cloud Object Storage Service (OBS) online. It is suitable for scenarios where a bucket contains no more than 3 TB of data or 5 million objects.

Constraints

For details, see OMS Notes and Constraints.

Precautions

• By default, if a task stays in the waiting state for more than 30 days, the system automatically puts the task into the Failed state and clears the AK/SK pair used by the task. To prevent task failures caused by blocking, do not create tasks too frequently.
• During the migration, the system automatically creates a temporary folder named OMS in the target bucket. Do not perform any operations on this folder, including but not limited to modifying, deleting, or adding data in the folder. Otherwise, the migration will be interrupted or fail.
• Keep both the tasks created in the past 24 hours and the tasks in the Waiting state under 1 million.

Procedure

Table 1 introduces the complete process of object storage migration with shared clusters.

Preparations

Prepare resources by referring to Table 2 before creating an object storage migration task with shared clusters.

Step 1: Select the Source and Target Ends

1. Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
2. In the navigation pane on the left, choose Storage Migration > Object Storage.
3. Choose Shared Cluster > Migration Tasks > Create Migration Task.
4. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK.
5. Determine whether to enable pre-migration evaluation.
   • If you disable it, pre-migration evaluation will not be performed.
   • If you enable it, specify the number of files to be migrated, the amount of data to be migrated, and the desired migration duration. Then click Start Evaluation. After the evaluation is complete, the pre-migration evaluation result is displayed under Start Evaluation.

6. Configure source parameters by referring to Table 3.

   Table 3 Source parameters

   Parameter	Description
   Data Source	The following source cloud service providers or data source types are supported: Huawei Cloud AWS Alibaba Cloud Microsoft Azure Tencent Cloud Google Cloud Qiniu Cloud QingCloud Baidu Cloud Kingsoft Cloud UCloud HTTP/HTTPS data sources
   Access Key ID	Enter the AK for accessing the source platform. It can contain a maximum of 100 characters.
   Secret Access Key	Enter the SK for accessing the source platform. The SK matches the AK, and they group into a pair.
   KMS Key	Disabled: The source key is not encrypted.
   	Enabled: The source key is encrypted. Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains: kms:cmk:get: Query key information. kms:dek:decrypt: Decrypt a data key. kms:dek:create: Create a DEK without plaintext. The authorization will succeed only if you have relevant permissions. Each IAM user only needs to perform the authorization once. Then, perform the following operations: Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported. NOTE: Enabling KMS encryption will incur fees. For details, see Pricing Details.
   Specify Source Bucket	Enable this option if your source account does not have the permissions required to list buckets but has the permissions to list objects in buckets, obtain object metadata, and obtain object content. Then specify the bucket or path to be migrated. For details, see Table 4.

   Table 4 Description of Specify Source Bucket

   If You...	Then...
   Enable Specify Source Bucket	Select the region where the source bucket is located from the Region drop-down list. In the Bucket text box, enter the bucket or the path to be migrated. The path must have the bucket name included. NOTE: OMS will check whether the source platform can be connected using the information you provided. If the source platform can be connected, the system will notify you.
   Disable Specify Source Bucket (This feature is disabled by default.)	Click the Bucket drop-down list or the refresh icon and select the bucket where data needs to be migrated. OMS will check whether the source platform can be connected using the information you provided. Once the source platform is connected, as long as your account has the necessary permissions to list buckets, all buckets in your account will appear in the Bucket drop-down list. NOTE: If the source account does not have the permissions required to list buckets, enable Specify Source Bucket.

7. Configure destination parameters by referring to Table 5.

   Table 5 Destination parameters

   Parameter	Description
   Access Key ID	Enter the AK for accessing Huawei Cloud. It can contain a maximum of 100 characters.
   Secret Access Key	Enter the SK for accessing Huawei Cloud. The SK matches the AK, and they group into a pair. It can contain a maximum of 100 characters.
   KMS Key	Disabled: The destination key is not encrypted.
   	Enabled: The source key is encrypted. Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains: kms:cmk:get: Query key information. kms:dek:decrypt: Decrypt a data key. kms:dek:create: Create a DEK without plaintext. The authorization will succeed only if you have relevant permissions. Each IAM user only needs to perform the authorization once. Then, perform the following operations: Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported. NOTE: Enabling KMS encryption will incur fees. For details, see Pricing Details.
   Bucket	Click the Bucket drop-down list or the refresh icon and select the Huawei Cloud OBS bucket for storing the migrated data. The bucket must be located in the region where you are using OMS.

8. After the source and target information is configured, click Next. The Set Task Parameters page is displayed.

Step 2: Set Task Parameters

1. Set parameters in the Source Configuration area according to the following table.

   Parameter	Description
   File/Folder	Click Select, and in the displayed dialog box, select the files or folders to be migrated, and click OK. You can click to remove a selected file or folder from the list. NOTE: If you do not select specific files or folders, all objects in the source bucket will be migrated. For each migration task, you can select up to 500 files and folders in a given directory. Due to OBS limitations, ensure that any object name at the target does not exceed 1,023 characters; otherwise, the migration task will fail.
   Object List	Generally, this method is used for incremental migration. You need to write the names of source objects to be migrated into a file and store the file in an OBS bucket on Huawei Cloud. OMS will migrate all objects specified in the file. Under List Bucket (Huawei Cloud), select the bucket where the object list is stored. NOTE: The listed buckets are obtained using the destination AK/SK pair you provided in 6. The bucket where the list file resides must be in the same region as the destination bucket. Otherwise, the object list file cannot be selected. Click Select next to Object List, select the object list file, and click OK. NOTE: An object list file cannot exceed 1,024 MB. The first object in the object list must exist, or the pre-verification will fail. An object list file must be a .txt file, and its metadata ContentType must be text/plain. An object list file must be in UTF-8 without BOM. Each line in an object list file can contain only one object name, and the object name must be URL-encoded if it contains special characters or spaces. Do not add invalid spaces to each line in the object list file. These spaces may cause migration failures because they may be mistakenly identified as object names. Each line in the object list file cannot be longer than 65,535 characters, or the migration will fail. The ContentEncoding metadata of the object list file must be left empty, or the migration will fail. Archived data can only be accessed and obtained by OMS after it is restored. If the object list file is archived, manually restore it before migration.
   Prefix	Enter the names or prefixes of objects to be migrated and click Add. The system will automatically list the matched objects. You can click to remove a name or prefix from the list. NOTE: If the files to be migrated are stored in the root directory of the source bucket, add their name prefixes directly. If the files are stored in a non-root directory, add their path prefixes in the format of <folder-name>/<prefix>. If no object names or prefixes are specified, all objects in the source bucket will be migrated.
   URL List	Generally, this method is used to migrate HTTP/HTTPS data. You need to write the URLs of the files to be migrated and their target object names into a URL list file and store the file in an OBS bucket on Huawei Cloud. OMS will migrate all files specified in the URL list file. Under List Bucket (Huawei Cloud), select the bucket where the object list is stored. NOTE: The listed buckets are obtained using the destination AK/SK pair you provided in 6. The bucket where the list file resides must be in the same region as the destination bucket. Otherwise, the object list file cannot be selected. For security purposes, if the migration source supports both HTTP and HTTPS, you are advised to use HTTPS for the migration. If HTTP is used, data may be eavesdropped on, tampered with, or vulnerable to man-in-the-middle or replay attacks. If the data to be migrated is not public and you do not want to make the data public, you are advised to use other migration methods. URL redirection is not supported. Click Select next to URL List, select the URL list file, and click OK. NOTE: A URL list file cannot exceed 1,024 MB. The URL list file must be a .txt file, and its metadata Content-Type must be text/plain. The URL list file must be in UTF-8 without BOM. Each line in the URL list file can contain only one URL and one destination object name. Each line in the URL list file cannot be longer than 65,535 characters, or the migration will fail. The ContentEncoding metadata of the URL list file must be left empty, or the migration will fail. The format of the first line in the URL list file must be correct, or the pre-verification will fail. In the URL list file, you must use a tab character (\t) to separate the URL and destination object name in each line. The format is [URL][Tab character][Destination object name]. Chinese and special characters in the source and destination object names must be URL-encoded. Please note that: The URL encoding must start from the second character after the domain name in a line. Do not encode the protocol header, domain name, or slash before or after the domain name. Otherwise, the format verification will fail. In each line, use a tab character (Tab key on the keyboard) to separate the URL and the destination object name. Do not use spaces. For example: http://xxx.xxx.xxx.xxx.com/doc/thefile2.txt doc/thefile2.txt http://xxx.xxx.xxx.xxx.com/the%20file.txt the%20file.txt http://xxx.xxx.xxx.xxx.com/the%20file2.txt the+file2.txt http://xxx.xxx.xxx.xxx.com/doc/thefile.txt doc/thefile.txt In the preceding examples, after the files represented by the URLs are copied to the destination bucket, the objects are named doc/thefile2.txt, the file.txt, the file2.txt, and doc/thefile.txt. Do not add invalid spaces to each line in the URL list file. These spaces may cause migration failures because they may be mistakenly identified as object names. URLs in the list file can be accessed using HEAD and GET methods.

2. Configure parameters in the Destination Configuration area.
   1. Save to Specified Prefix
      • If you choose to disable this feature, the names of objects remain unchanged after the migration.
      • If you choose to enable this feature, you can specify a prefix to rename or redefine the paths for objects migrated to the destination bucket.
        

        To learn how to use Specify Prefix, see Adding a Name Prefix or Path Prefix to Migrated Objects.

   2. Object Storage Class

      Choose the storage class that your data will be migrated to. For details about storage classes, see How Do I Choose Storage Classes?

3. (Optional) Click before Advanced Settings to expand the advanced options. Configure advanced options by referring to Table 6.

   Table 6 Advanced settings

   Parameter	Description
   Object Metadata	Determine whether to migrate metadata. If you select Do not migrate, OMS will only migrate the Content-Type and Content-Encoding metadata. If you select Migrate, OMS will migrate supported metadata. The following metadata types are supported: Content-Type, Content-Encoding, Content-Disposition, Content-Language, Cache-Control, Expires, WebsiteRedirectLocation, Storage-Class, and custom metadata. NOTICE: Only Chinese characters, English characters, digits, and hyphens (-) can be migrated. Other characters cannot be migrated. Even if you choose not to migrate metadata, the Content-Type and Content-Encoding metadata will still be migrated to ensure that the migration runs properly.
   Incremental Data	Determine whether to filter source data to be migrated by time. If you do not need to filter source data by time, disable this option. If you need to filter source data by time, enable this option and specify a date and time. Then only source data modified after the specified date and time will be migrated.
   Failed Object Record	Determine whether to record the objects that fail to be migrated. If you disable this option, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be transmitted again. If you enable this option, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, only the failed objects need to be migrated again. NOTE: For a migration task, whether it is contained in a task group or not, any objects that fail to be migrated are recorded in a list named after the task name. This list is then uploaded to the oms/failed_object_lists/ directory of the destination bucket. A maximum of 100,000 failed objects can be recorded in a list file. If more than 100,000 objects fail to be migrated in a migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again. This file contains information about the failed migration task, including the failure cause and time (in local UTC), URL-encoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs. This file is retained in the destination bucket even after the objects in the file are successfully re-transferred. If you do not need the file, delete it manually.
   Archive Data Restoration	Archived data must be restored before it can be migrated. You can enable this option if your source cloud service provider supports automatic restoration of archived data. If you enable this option, the system automatically restores and migrates archived objects in the migration task. If an archived object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. NOTE: This function is applicable to the following clouds: Huawei Cloud (COLD), Alibaba Cloud (Archive), Kingsoft Cloud (ARCHIVE), Tencent Cloud (ARCHIVE), and UCloud (ARCHIVE). It takes at least a minute to restore an archived object. The larger the archived object is, the longer it takes to restore it, but if an archived object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see the documentation of your source cloud service provider. Two types of fees are generated during archived data restoration: fees for API calls and fees for archived object restoration. These fees are billed by the source cloud service provider. After restoration is complete, archived objects remain in the restored state for a certain period. During this period, they can be downloaded or read. Once the period expires, the objects must be restored again. For details about how long they stay in the restored state, see the documentation of the cloud service provider. The copies generated for restoring archived objects will be stored for three days by default. If there are more than 3 TB of data or more than 3 million objects in a source bucket, create a migration task group to migrate data in case the migration fails because the restored data is archived again. Pausing or restarting a migration task will result in restoration of all archived data again, regardless of whether it has been restored or not. This process will require additional time and incur extra costs. After objects are migrated from the source platform to Huawei Cloud, they are in the same storage class as the destination bucket. If you do not enable this option, the system directly records archived objects in the list of objects that fail to be migrated, skips them, and continues to migrate other objects in the migration task.
   SMN Notification	Determine whether to use SMN to get notifications about migration results. If you do not enable this option, no SMN messages will be sent after the migration is complete. If you enable this option, SMN messages will be sent after the migration is complete. Enabling this option indicates that you agree to assign OMS the permissions to publish messages to the selected topic through an agency (SMN FullAccess). You can delete the authorization at any time. Perform the following steps: Click Add next to the SMN Topic box. Select a topic for publishing SMN messages and click OK. Click Add next to the SMN Template box. If this parameter is specified, SMN messages will be sent using the specified template. The template must use the default protocol. Select a template for publishing SMN messages and click OK. Configure Trigger Condition. You can select one of the following options or both: Migration succeeded: An SMN message will be sent if the migration task succeeds. Migration failed: An SMN message will be sent if the migration task fails. If you experience trouble in using SMN, see What Do I Do If SMN Does Not Work Properly?
   Limit Traffic	Set the maximum bandwidth for the migration task during different periods. If you do not need to limit the migration traffic, disable this option. If you need to limit the migration traffic, enable this option. Specify Start Time, End Time, and Maximum Bandwidth. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited beyond this period. NOTE: The maximum bandwidth ranges from 1 MB/s to 200 MB/s. Time periods in different rules cannot overlap. For example, if there is a rule configured for the period from 8:00 to 12:00, you cannot configure rules for any overlapping periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00. Start and end times must be on the same day, with the start time preceding the end time. For example, the time period from 23:00 to 01:00 of the next day is not allowed. A maximum of five rules can be added. The time is the local standard time of the region you are migrating to.
   Object Overwrite	Overwrite if source newer or different size: This is the default setting. If a source object has the same name as the paired destination object, the system compares the sizes and last modification time of the source and destination objects to determine whether to overwrite the destination object. The encryption status of the source and destination objects must be the same. If a source object was last modified more recently than or is not as large as its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. Never overwrite: The system always skips source objects and keeps their paired destination objects. Overwrite if different CRC64 checksum: This option is only available for migration within Huawei or from Alibaba or Tencent Cloud. If a source object has a CRC64 checksum different from the paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. If either of them lacks a CRC64 checksum in the metadata, their sizes and last modification times are checked. Always overwrite: The system always allows source objects to overwrite their paired destination objects.
   Consistency Check	By size and last modification time: This is the default setting. The system checks object consistency with object size and last modification time. If a source object is as large as but was last modified earlier than its paired destination object, the system considers that the source object does not need to be migrated or has already been migrated successfully. By CRC64 checksum: This option is only available for migration within Huawei or from Alibaba Cloud or Tencent Cloud. If a source object and its paired destination object have CRC64 checksums, the checksums are checked. Otherwise, their sizes and last modification times are checked.
   Priority	If there are a large number of tasks, you can set task priorities to control which tasks take precedence over other tasks.
   Description	Enter a brief description about the migration task. Enter a maximum of 255 characters.

Step 3: Confirm the Information

1. After the migration task is configured, click Next to confirm the information.
2. Confirm the information is correct and click Start Migration. Read the prompted information. After you confirm, click OK. A message is displayed, indicating that the task is submitted.

Step 4: Track the Migration Task

1. Go back to the task list and check the created task.
   Figure 1 Task list
   

2. Click the task name or ID to check the number of migrated objects, total number of objects, task progress, migration speed, source and destination information, and migration task settings. You can also perform the following operations:
   • Add, edit, or delete traffic limiting rules to control cluster traffic based on your requirements.
     Click under Traffic Limiting.

     • To add a rule, click Add Rule and set Start Time, End Time, and Rate Limit.
     • To modify a rule, modify Start Time, End Time, and Rate Limit.
     • To delete a rule, click Delete in the Operation column.
       

       • The maximum bandwidth ranges from 1 MB/s to 200 MB/s.
       • Time periods in different rules cannot overlap.

         For example, if there is a rule configured for the period from 8:00 to 12:00, you cannot configure rules for any overlapping periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00.

       • Start and end times must be on the same day, with the start time preceding the end time.

         For example, the time period from 23:00 to 01:00 of the next day is not allowed.

       • A maximum of five rules can be added.
       • The time is the local standard time of the region you are migrating to.

Managing Migration Tasks

In the migration task list, you can start, stop, restart, or delete a migration task by referring to Table 7.