Creating a Migration Task Group

Scenarios

If a source bucket contains more than 3 TB of data or more than 5 million objects, you can create a migration task group to migrate the source data quickly. The system will group the source objects into multiple sub-tasks for concurrent migrations.

To migrate data faster, you are advised to use dedicated clusters in MgC. MgC enables you to migrate data using dedicated, scalable migration clusters and up to 20 Gbit/s of bandwidth.

Constraints

• Fewer than 1,000,000 migration tasks have been created in your account in the past 24 hours.
• There are fewer than 1,000,000 migration tasks in the Waiting state in your account.
• You have created fewer than 300 migration task groups within the past 24 hours.

For more constraints on object storage migration tasks, see Notes and Constraints.

Precautions

• OMS makes API calls to the object storage services on source platforms during the migration. These API calls may generate costs on those platforms and you will be responsible for those costs based on those source platform billing structures.
• By default, if a task stays in the waiting state for more than 30 days, OMS automatically puts the task into the Failed state and clears the AK/SK pair used by the task. To prevent task failures caused by blocking, do not create tasks too frequently.
• During the migration, the system automatically creates a temporary folder named OMS in the destination bucket. Do not perform any operations on this folder, including but not limited to modifying, deleting, or adding data in the folder. Otherwise, the migration will be interrupted or fail.

Procedure

Table 1 introduces the complete process of creating an object storage migration task group.

Step 1: Performing Pre-Migration Evaluation

1. Log in to the OMS console.
2. Click in the upper left corner and select a region from the drop-down list.
3. In the navigation pane, choose Migration Task Groups. The Migration Task Groups page is displayed.
4. On the Migration Task Groups page, click Create Migration Task Group in the upper right corner.
5. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK. The Create Migration Task Group page is displayed.
6. Determine whether to enable pre-migration evaluation.
   • If you disable it, pre-migration evaluation will not be performed.
   • If you enable it, specify the number of files to be migrated, the amount of data to be migrated, and the desired migration duration. Then, click Start Evaluation. After the evaluation is complete, the result is displayed under Start Evaluation.

Step 2: Selecting Source and Destination

1. In the Select Source area on the Select Source/Destination page, set source parameters based on Table 2.

   Table 2 Source parameters

   Parameter	Setting
   Data Source	Select Tencent Cloud.
   APP_ID	Enter the APP_ID of your Tencent Cloud account. NOTE: You can view the APP_ID on the account information page of the Tencent Cloud console.
   Access Key ID	Enter the SecretId of your Tencent Cloud account to which the source bucket belongs.
   Secret Access Key	Enter the SecretKey of your Tencent Cloud account to which the source bucket belongs.
   KMS Key	Disabled: The source key is not encrypted.
   	Enabled: The source key is encrypted. Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains: kms:cmk:get: Query key information. kms:dek:decrypt: Decrypt a data key. kms:dek:create: Create a DEK without plaintext. If you have the permission to grant access, grant KMS access permissions directly. After KMS access permissions are granted, follow-up operations do not require the permissions to be granted again. Then, you can perform the following operations: Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported. NOTE: Enabling KMS encryption will incur fees. For details, see Pricing Details.
   Specify Source Bucket	Disable Specify Source Bucket. This feature is disabled by default. Click the Bucket drop-down list or the refresh icon and select the bucket where data needs to be migrated. OMS will check whether the source platform can be connected using the information you provided. Once the source platform is connected, as long as your account has the necessary permissions to list buckets, all buckets in your account will appear in the Bucket drop-down list. NOTE: If the source account does not have the permissions required to list buckets, enable Specify Source Bucket.
   	Enable Specify Source Bucket. Select the region where the source bucket is located from the Region drop-down list. In the Bucket text box, enter the bucket or the path to be migrated. The path must include the bucket name. The entered bucket name should not contain APP_ID.

2. In the Select Destination area, set destination parameters based on Table 3.

   Table 3 Destination parameters

   Parameter	Description
   Access Key ID	Enter the AK for the Huawei Cloud account to which the destination bucket belongs.
   Secret Access Key	Enter the SK for the Huawei Cloud account to which the destination bucket belongs.
   KMS Key	Disabled: The destination key is not encrypted.
   	Enabled: The destination key is encrypted. Before enabling this function, authorize OMS to access KMS. After that, a custom policy is granted, which contains: kms:cmk:get: Query key information. kms:dek:decrypt: Decrypt a data key. kms:dek:create: Create a DEK without plaintext. If you have the permission to grant access, grant KMS access permissions directly. After KMS access permissions are granted, follow-up operations do not require the permissions to be granted again. Then, you can perform the following operations: Select a key from the KMS key drop-down list. Alternatively, create one by referring to Creating a Custom Key. Currently, only the AES_256 and SM4 key algorithms are supported. NOTE: Enabling KMS encryption will incur fees. For details, see Pricing Details.
   Bucket	Click the Bucket drop-down list or click Refresh and select the Huawei Cloud OBS bucket for storing the migrated data. The bucket must be located in the region where you are using OMS.

3. Click Next in the lower right corner. The Set Task Parameters page is displayed.

Step 3: Setting Migration Task Group Parameters

1. In the Source Configuration area, select a migration method based on your requirements. Table 4 describes specific migration methods.

   Table 4 Migration methods

   Migration Method	Description	Operation
   Prefix	Enter an object name prefix. All objects whose names start with the prefix in the bucket will be migrated. If no prefixes are specified, all objects in the source bucket will be migrated. If the files to be migrated are stored in the root directory of the source bucket, enter their prefixes directly. If the files are stored in a non-root directory, add their directories and their prefixes in the format of Folder name/Prefix. The objects to be migrated are recorded in .txt files, and each migration task has a list file. The storage path of the files is oms/taskgroup/ in the destination bucket. These object list files will be retained after the migration, and you can manually delete them.	Enter the name or name prefix of the object to be migrated in the text box. Click Add. The system will automatically list the matched objects. You can click to delete the objects that do not need to be migrated from the list.
   Object List	Write the names of the source objects to be migrated into a .txt file, one name per line, and upload the file to a Huawei Cloud OBS bucket. The bucket must be in the same region as the destination bucket; otherwise, the list files cannot be selected. OMS will read the object list and migrate all objects defined in the list. An object list file must meet the following requirements: The file cannot exceed 1,024 MB. The first object in the file must exist, or the pre-verification will fail. The file must be a .txt file, and its metadata Content-Type must be text/plain. The file must be in UTF-8 without BOM. Each line in the file can contain only one object name. The object name must be URL-encoded if it contains special characters or spaces. Do not add invalid spaces to each line in the file. These spaces may cause migration failures because they may be mistakenly identified as object names. Each line in the file cannot be longer than 65,535 characters, or the migration will fail. The Content-Encoding metadata of an object list file must be left empty, or the migration will fail. Archived data can only be accessed and obtained by OMS after it is restored. If the object list file is archived, manually restore it before migration.	In the List File Bucket (Huawei Cloud) drop-down list, select the bucket where the list file is stored. Click Select next to Object List. The Select Object List dialog box is displayed. Select the object list file and click OK.

2. In the Destination Configuration area, set parameters based on Table 5.

   Table 5 Destination parameters

   Parameter	Description
   Specify Prefix	If you choose to disable this feature, the names of objects remain unchanged after the migration. If you choose to enable this feature, you can specify a prefix to rename or redefine the paths for objects migrated to the destination bucket. For details about how to set the Specify Prefix parameter, see Using a Specified Prefix for Migration.
   Object Storage Class	Choose the storage class of objects after they are migrated to the destination OBS bucket. For details about storage classes, see How Do I Choose Storage Classes?

3. Click next to Advanced Options to expand the advanced options. Configure the parameters based on Table 6.

   Table 6 Advanced options

   Parameter	Description
   Object Metadata	Determine whether to migrate metadata. If you select Do not migrate, OMS will only migrate the Content-Type and Content-Encoding metadata to ensure that the migration runs properly. If you select Migrate, OMS will migrate supported metadata. Only Chinese characters, English letters, digits, and hyphens (-) can be migrated. Other characters cannot be migrated. The following metadata types are supported: Content-Type, Content-Encoding, Content-Disposition, Content-Language, Cache-Control, Expires, WebsiteRedirectLocation, Storage-Class, and custom metadata.
   Selective Migration	Determine whether to filter source data to be migrated by time. If you do not need to filter source data by time, disable this option. If you need to filter source data by time, enable this option and specify a date and time. Then only source data modified after the specified date and time will be migrated.
   Failed Object Record	Determine whether to record the objects that fail to be migrated. If you disable this option, the system will not record the objects that fail to be migrated. If the migration fails and needs to be retried, all objects in the source bucket need to be migrated again. If you enable this option, the system will record the objects that fail to be migrated. If the migration fails and needs to be retried, you can choose to only migrate failed objects again. NOTE: For each migration task, whether or not it belongs to a task group, the objects that fail to be migrated are recorded in a list named after the task name. This list is then uploaded to the oms/failed_object_lists/ directory in the destination bucket. A maximum of 100,000 failed objects can be recorded in a list file. If more than 100,000 objects fail to be migrated in a migration task, you are advised to rectify the fault based on the existing failed object list and perform the migration again. This file contains information about the failed migration task, including the failure cause and time (in local UTC), URL-encoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs. This file is retained in the destination bucket even after the objects in the file are successfully re-transferred. If you do not need the file, delete it manually.
   Archive Data Restoration	Archived data must be restored before it can be migrated. You can enable this option if your source cloud service provider supports automatic restoration of archived data. This function is applicable to the following clouds: Huawei Cloud (COLD), Alibaba Cloud (Archive), Kingsoft Cloud (ARCHIVE), Tencent Cloud (ARCHIVE), and UCloud (ARCHIVE). If you do not enable this option, the system directly records archived objects in the list of objects that fail to be migrated, skips them, and continues to migrate other objects in the migration task. If you enable this option, the system automatically restores and migrates archived objects in the migration task. If an archived object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. NOTE: It takes at least a minute to restore an archived object. The larger the archived object is, the longer it takes to restore it, but if an archived object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see the documentation of your source cloud service provider. Two types of fees are generated during archived data restoration: fees for API calls and fees for archived object restoration. These fees are billed by the source cloud service provider. After the restoration is complete, archived objects will be in the restored state for a certain period of time. During this period, objects can be downloaded or read. After this period expires, archived objects need to be restored again for migration. For details about how long they stay in the restored state, see the documentation of the cloud service provider. The copies generated for restoring archived objects will be stored for three days by default. If there are more than 3 TB of data or more than 3 million objects in a source bucket, create a migration task group to migrate data in case the migration fails because the restored data is archived again. Pausing or restarting a migration task will cause all archived data to be restored again, even if it was already restored. This operation will take additional time and generate extra costs. After objects are migrated from the source platform to Huawei Cloud, they are in the same storage class as the destination bucket.
   SMN Notification	Determine whether to use SMN to get notifications about migration results. If you do not enable this option, no SMN messages will be sent after the migration is complete. If you enable this option, SMN messages will be sent after the migration is complete. Enabling this option means you agree to authorize OMS to publish messages to the selected topic through an agency (SMN FullAccess). You can delete the authorization at any time. Then, perform the following steps: Click Add next to the SMN Topic box. Select a topic for publishing SMN messages and click OK. Click Add next to the SMN Template box. If this parameter is specified, SMN messages will be sent using the specified template. The template must use the default protocol. Select a template for publishing SMN messages and click OK. Configure Trigger Condition. You can select one of the following options or both: Migration succeeded: A success notification will be sent via SMN if the migration task succeeds. Migration failed: A failure notification will be sent via SMN if the migration task fails. NOTE: If you experience trouble in using SMN, see What Do I Do If SMN Does Not Work Properly?
   Traffic Limit	Set the maximum bandwidth for the migration task during different periods. If you do not need to limit the migration traffic, disable this option. If you need to limit the migration traffic, enable this option. Specify Start Time, End Time, and Maximum Bandwidth. For example, if you set the start time to 08:00, end time to 12:00, and the maximum bandwidth to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited beyond this period. NOTE: The maximum bandwidth ranges from 1 MB/s to 200 MB/s. When creating multiple rate limiting rules, ensure that the time periods do not overlap. For example, if there is a rule configured for the period from 8:00 to 12:00, you cannot configure rules for any overlapped periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00. Start and end times must be on the same day, with the start time preceding the end time. For example, the time period from 23:00 to 01:00 on the following day is not allowed. A maximum of five rules can be added for a migration task group. The time is the local standard time of the region you are migrating to.
   Object Overwrite	Overwrite if source newer or different size: This is the default setting. If a source object has the same name as the paired destination object, the system compares the sizes and last modification time of the source and destination objects to determine whether to overwrite the destination object. The encryption status of the source and destination objects must be the same. If a source object was last modified more recently than or is not as large as its paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. Never overwrite: The system always skips source objects and keeps their paired destination objects. Overwrite if different CRC64 checksum: This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud. If a source object has a CRC64 checksum different from the paired destination object, the source object will overwrite the destination object. Otherwise, the source object will be skipped. If one of them does not have a CRC64 checksum, their sizes and last modification times are checked. Always overwrite: The system always allows source objects to overwrite their paired destination objects.
   Consistency Check	By size and last modification time: This is the default setting. The system checks object consistency with object size and last-modification time. If a source object and its paired destination object have the same size and the destination object's last-modification time is not earlier than the source object's, the system considers the object unnecessary to migrate or already successfully migrated. By CRC64 checksum: This option is only available for migration within Huawei Cloud or from Alibaba Cloud or Tencent Cloud. If a source object and its paired destination object have CRC64 checksums, the checksums are checked. Otherwise, their sizes and last modification times are checked. Not required: This option is only available for migration of HTTP/HTTPS data. This option takes effect for source objects whose sizes cannot be obtained using the Content-Length field in the standard HTTP protocol. These source objects will overwrite their paired destination objects directly. If the size of a source object can be obtained, its size and last modification time will be checked.
   Description	Enter a brief description about the migration task. A maximum of 255 characters is allowed.

4. Click Next in the lower right corner of the page. The Confirm Information page is displayed.

Step 4: Starting and Tracking the Migration Task Group

1. On the Confirm Information page, confirm the migration parameter settings and click Migrate Now in the lower right corner. To update the parameter settings, click Previous to return to the corresponding page.
2. Read the information in the Confirm dialog box and click OK. If the message Request submitted is displayed, the migration task group has been created successfully and the system will automatically start the migration.
3. Click Back to Task List to view and manage the migration task group. For details, see Viewing a Migration Task Group and Managing a Migration Task Group.