Help Center/ Migration Center/ User Guide/ Application Migration Projects/ Storage Migration/ Batch Object Storage Migration
Updated on 2026-05-22 GMT+08:00
View PDF
Share

Batch Object Storage Migration

What Is Batch Object Storage Migration?

This solution provides dedicated migration clusters and private lines to support batch creation of object storage data migration tasks, enabling efficient migration of object storage data. It is designed to migrate buckets in batches. Using exclusive clusters will incur fees. For details, see Billing.

Warnings

When creating a task, there are four overwriting policies for objects with the same name at the source and target: If older or different size, Never, Always, and If different CRC64 checksum. If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.

Notes and Constraints

  • A migration task can be used to migrate buckets from the same source platform and to the same target region.
  • A source bucket can be included in multiple migration plans.
  • In a migration task, a bucket can have only one destination prefix but can have multiple source prefixes.
  • Each user is allowed to migrate a maximum of 20 buckets at the same time in a single migration task.

For more constraints, see Constraints on Storage Migration.

Procedure

Table 1 introduces the complete process of batch object storage migration.

Table 1 Process of batch object storage migration

Description

Description

Preparations
  • Before using MgC, prepare a HUAWEI ID or an IAM user that can access MgC, and grant them relevant permissions.
  • Obtain the AK/SK pair of the source account for resource collection.
  • Create a server migration project on the MgC console.
  • Create a target bucket for migration.
  • Create a migration cluster to improve migration efficiency.

Step 1: Collect Source Resources

Discover and collect object storage resources using Online Survey.

Step 2: Create a Migration Task

Create a batch object storage migration task. At this step, you will create a task template for configuration later.

Step 3: Configure the Migration Task

Select source resources, set migration parameters, configure the migration cluster, and confirm the settings.

Step 4: Track the Migration Task

Start the migration task and check the migration status, phases, and steps.

Preparations

Prepare resources by referring to Table 2 before creating a batch object storage migration task.

Table 2 Preparations

Item

Requirement

Preparing accounts and permissions
  1. Sign up for a HUAWEI ID, enable Huawei Cloud services, and complete real-name authentication. If you have already enabled Huawei Cloud services and completed real-name authentication, skip this step.
    1. Visit Huawei Cloud and click Sign Up.
    2. Sign up for a HUAWEI ID and enable Huawei Cloud services.
    3. (Optional) Complete real-name authentication. Real-name authentication is the process of submitting valid identity information (such as a passport or business license) for verification when you use cloud services in the Chinese mainland. There are individual real-name authentication and enterprise real-name authentication.

      According to relevant laws and regulations, unauthenticated accounts cannot purchase cloud services in the Chinese mainland and your account and rights may not be protected. For more information, see Real-Name Authentication Overview.

      To complete real-name authentication, refer to:
  2. Create an IAM user and grant permissions to the user. Skip this step if you use a HUAWEI ID (administrator account).
    1. Create an IAM user. For details, see Creating an IAM User or Creating an IAM User (New Version).
    2. Create a custom policy and grant permissions to the IAM user. For details, see Using Roles or Policies to Assign Custom Permissions to IAM Users or Using Identity Policies to Assign Custom Permissions to IAM Users (New Version). For details about how to grant permissions, see Assigning Permissions to an IAM User or Assigning Permissions to an IAM User (New Version).
  3. (Optional) Obtain the AK/SK of the target account.

    The AK/SK of the target account is required for authentication, especially when a destination OBS bucket is used. For details about how to obtain the AK/SK of the target account, see:

    Creating an Access Key or Access Keys (New Version).

Obtaining the AK/SK of the source account

The AK/SK of the source account is required for authentication when collecting resources. For details about how to obtain the AK/SK of the source account, see:

How Do I Obtain Cloud Platform Credentials (AK/SK Pairs)?

Creating an application migration project

Create a project to separate and manage migration resources. For details, see Creating an Application Migration Project.

Creating a target bucket

Create a target bucket in advance to ensure that the storage workflow runs properly. For details, see Creating a Bucket.

Creating a migration cluster

A migration cluster is a core component for storage workflows. It allows you to create and manage migration nodes, list nodes, and deploy and upgrade migration plugins, ensuring smooth operations of storage workflows. For details, see Creating a Migration Cluster.

Step 1: Collect Source Resources

  1. Sign in to the MgC console. In the navigation pane, under Project, select your application migration project from the drop-down list.
  2. In the navigation pane, click Resource Survey.
  3. Under Online Discovery, click Cloud Discovery.

    Figure 1 Cloud discovery

  4. Set collection task parameters according to Table 3.

    Table 3 Parameters for creating a cloud discovery task

    Region

    Parameter

    Description

    Task Settings

    Source Platform

    Select Huawei Cloud.

    Collection Method

    Credential: In the drop-down list, select the AK/SK of the account to which the source server belongs. For details, see Managing Credentials. Alternatively, add one by performing the following steps:

    1. Click Create Credential and enter a name.
    2. Set Authentication Mode to AK/SK and enter the AK and SK of the source account.
    3. Click Save.

    Agency: A cloud service agency is required to assign MgC the permissions required to access resources in your current account. For details about the agency permissions, see Agency Permissions.

    If the Agency Status is Agency not created, click Create Agency. On the displayed page, click OK.

    Region

    Select the region where the source servers are located. You can select multiple regions.

    Resource Type

    Select Servers from the drop-down list.

    Basic Information

    Task Name

    The default name format is <task type>-<current-time>. You can also specify a name.

    Task Description (Optional)

    Describe the task.

    Application (Optional)

    Select the application that you want to group the discovered resources into. This is recommended for the assessment and migration of a small number of resources. If no applications are available, perform the following steps to create one:

    1. Click Create Application. Enter an application name and description.
    2. Set the scenario as required.
    3. Set the environment as required.
    4. Select the region you want to migrate to.
    5. Click OK.

  5. Click Confirm. After the online discovery task is created, MgC starts discovering source resources.

    • If the task succeeds, return to the Resource List page and check the list of discovered source resources and their details.
    • If the task fails, click the task name to check the data source that fails to be collected. Hover over to the collection status to check the failure cause.

Step 2: Create a Migration Task

  1. In the navigation pane on the left, choose Storage Migration > Bulk Migration.
  2. Choose Object Storage > Create Migration Task.
  3. Use the default task name or create one. Enter the task description (optional).
  4. Click OK to complete the task creation. On the displayed page, configure the migration task by referring to Step 3: Configure the Migration Task.

Step 3: Configure the Migration Task

This step includes four phases.

  1. On the Configure Source/Target page, select the source cloud platform from the Source Platform drop-down list. A migration plan can be used to migrate buckets from the same source platform and to the same target region.
  2. From the Target Region drop-down list, select the region where the target buckets are located.
  3. In the Source Buckets area, add buckets in one of the following ways. A maximum of 20 buckets can be added.

    • Source bucket information collected: Click Select Source Bucket. Select a bucket to be migrated, click Modify in the Operation column, configure endpoints, and click Confirm.
    • Source bucket information not collected: Click Add Source Bucket. Set parameters based on Table 4.
      Table 4 Parameters for adding a source bucket

      Parameter

      Description

      Source Region

      Select the region where the source bucket is located.

      Source bucket name

      Name of the source bucket. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed. The name must start with a letter or digit, and it must not start or end with a period.

      NOTE:
      • The name you enter here must match the corresponding name on the source platform. Otherwise, the migration fails.
      • The bucket's region and name cannot be the same as those of an existing bucket. Otherwise, the operation will fail after you click Verify and Save.

      Address

      Enter the endpoint of the region where the source bucket is located.

      For example, if Source Platform is set to Alibaba Cloud OSS and the source bucket is located in the CN East 1 (Hangzhou) region, enter oss-cn-hangzhou.aliyuncs.com.

  4. In the Buckets area, set the bucket type, source credential, target bucket, and target credential in each row. If the source platform is Tencent Cloud, set appId. Click Save in the Operation column. You can also click Save All after setting multiple records. After the settings are saved, batch modification is supported.
  5. After you configure all the buckets and save the settings, click Next.
  1. In the Migration Settings area, set the migration method, migration scope, and (optional) target prefix in each row. Refer to Table 5 to learn how to configure Migration Method and Migration Scope.

    Table 5 Migration methods and scope

    Migration Method

    Description

    Migration Scope

    Full migration

    Migrates all data in a source bucket or folder.

    N/A

    Prefix migration

    Migrates objects matched with the specified names or prefixes.

    If no source prefixes are specified, a full migration will go. There are two methods to specify source prefixes.

    • Import Source Prefixes: Import a TXT file that lists prefixes.
    • Add Source Prefixes: Enter prefixes.

    List migration

    Migrates objects recorded in the list files. The list files must be stored in the target bucket.

    (Mandatory) Enter the path for storing the list files in the Migration Scope box and end the path with a slash (/).

  2. (Optional) In the Source Settings area, select Migrate Metadata and Restore Archive Data.

    If the source is Alibaba Cloud OSS or Tencent Cloud COS, you can enable Download Data from CDN. If CDN authentication is required, you also need to select an authentication type and enter an authentication key.

  3. In the Target Settings area, set the parameters based on Table 6.

    Table 6 Migration parameters

    Parameter

    Sub-parameter

    Description

    Overwrite Existing

    If older or different size
    • The system replaces existing target files if they are older than or differ in size from their source counterparts. Files with matching modification times and sizes remain unchanged and are skipped during migration.
    • The system verifies folders after their content is migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source.

    Never

    Files existing at the target will never be overwritten.

    WARNING:

    If you choose Never, restarting a migration after an interruption or pause may lead to incomplete data migration, even though the task may appear successful. This could impact data integrity, so use the Never option with caution.

    Always

    Files existing at the target will always be overwritten.

    If different CRC 64 checksum

    If a source object has a CRC64 checksum different from the paired target object, the source object will overwrite the target one. Otherwise, the source object will be skipped during the migration. If either of them does not have a CRC64 checksum, their sizes and last modification times are checked.

    NOTE:
    • This option is only available for migration within Huawei Cloud or from Alibaba Cloud, Tencent Cloud, or Azure.
    • Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.

    Target Storage Class

    Standard

    For details about storage classes, see How Do I Choose Storage Classes?

    NOTE:

    CRC64-based consistency verification is not available for Archive and Deep Archive storage classes. Even if you choose the CRC64-based verification method, the system automatically uses the object size and last modification time to verify data consistency.

    Infrequent Access

    Archive

    Deep Archive

    Same as source

  4. In the Consistency Verification area, select a consistency check method.

    • Size and last modified: With this default setting, the system verifies object consistency with object size and last modification time.
    • CRC64 checksum: The system checks data consistency by comparing the CRC64 values in the metadata of source and target objects. If a source object or the paired target object does not have a CRC64 checksum, the OMS-calculated CRC64 value is used for verification. CRC64 verification may generate extra public traffic and request costs. For details, see Consistency Verification.
      • This option is only available for migration within Huawei Cloud or from Alibaba Cloud, Tencent Cloud, or Azure.
      • Using this option requires that target OBS buckets be whitelisted for the CRC64 feature.

  5. In the Migration Settings area, set parameters according to Table 7.

    Table 7 Migration settings

    Parameter

    Description

    Concurrent Subtasks

    User-defined. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, there can be up to 20 subtasks.

    (Optional) Flow Control

    Set the maximum bandwidth for the migration task during different periods.

    • If you do not select this option, migration traffic will not be limited.
    • If you select this option, you can limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit.
      For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited outside this period.
      NOTE:
      • The bandwidth limit ranges from 1 MB/s to 1,048,576 MB/s.
      • Time periods in different rules cannot overlap.

        For example, if there is a rule added for the period from 8:00 to 12:00, you cannot configure rules for any overlapped periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00.

      • The start time of a rule cannot be later than the end time.

        For example, the time period from 23:00 to 01:00 of the next day is not allowed.

      • A maximum of five rules can be added.
      • The time is the local standard time of the region you are migrating to.

  6. (Optional) Configure advanced options based on Table 8.

    Table 8 Advanced options

    Parameter

    Description

    Enable KMS Encryption
    • If you do not select this option, whether migrated data will be encrypted in the target buckets depends on the server-side encryption setting of the buckets.
    • If you select this option, all migrated objects will be encrypted before they are stored to the target bucket.
    NOTE:
    • Using KMS to encrypt migrated data may slow down the migration speed by about 10%.
    • This option is only available when KMS is supported in the region you are migrating to.

    Filter Source Data

    Use exclude patterns, include patterns, and time ranges to filter files to be migrated. For details about the filters, see Source Data Filters.

    Send SMN Notifications

    Determine whether to use SMN to get notifications about migration results.

    • If you do not select this option, no SMN messages will be sent after the migration is complete.
    • If you select this option, after the migration is complete, SMN messages will be sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.

    Migrating Incremental Data
    • If this option is not selected, incremental migration will not be performed.
    • If this option is selected, configure the overwriting policy and the execution mode for incremental migration. For details, see Configuring Incremental Migration Settings.

  7. After the migration settings are configured, click Next to open the Configure Migration Cluster page.
  1. In the Configuration Method area, select a method based on whether a cluster is available. If a cluster is available, choose Use existing cluster. If no migration clusters are available, choose Explore recommendations and create new cluster to assess the appropriate cluster size. You can assess how large of a migration cluster is required for the migration and then create a migration cluster with the recommended specifications. Alternatively, you can skip this step and use an existing migration cluster. For details about how to create a cluster, see Creating a Migration Cluster.
  2. In the Migration Cluster area, select the cluster to be used. To modify the cluster, click Modify in the Operation column and modify the cluster by referring to Managing a Migration Cluster.
  3. After the migration cluster is configured, click Next to open the confirmation page.
  1. After completing operations from Phase 1: Configure Migration Resources to Phase 3: Configure the Migration Cluster, you will be directed to the Confirm page. Check the resource and migration settings carefully to ensure that they meet your requirements. To update the settings, click Previous to return to the corresponding stage for modification.
  2. On the Confirm page, verify that the resource and migration settings are correct and that all quotas meet the requirements. Then, click Finish. The Start Migration Task dialog box is displayed. The migration task has been created. Determine whether to start the migration immediately.

    • If you want to start the migration immediately, click Confirm to run the task.
    • If you want to save the migration settings without starting the migration, click Cancel. The task status will change to To be started. To run the task, click Run in the Operation column to manually start the migration.

  3. (Optional) Add a phase or step.

    Adding checkpoints to migration tasks helps you control the migration process. For details, see Adding a Pre-/Post-Migration Stage and Step (Checkpoints and Scripts).

Step 4: Track the Migration Task

In the migration task list, click the task name to go to the Task Details page. Check the task progress and perform the following operations on it.
  • Checking the resource status: Click a resource name. On the Resource Details page displayed on the right, you can review the basic information, migration status, and migration progress of the resource.
  • Confirming steps: Some resources require manual confirmation. In this case, click Confirm in the Migration Status column or in the Operation column to continue the migration.

    To check the migration phases and steps included in a migration task, click View Task Template in the upper right corner of the task details page.

Reconfiguring a Task

Tasks that are to be started, paused, failed to be deleted, or successful can be reconfigured. The task status will change to Configuring. All configured resources and their progress will be deleted permanently.

  1. In the migration task list, choose More > Reconfigure in the Operation column of a migration task.
  2. Enter YES and click OK.
  3. Wait until the task status changes to Configuring. Click Configure to reset the task.

Deleting Resources from a Migration Task

Resources in the Failed, Paused, or Completed state can be deleted. If all resources in a migration task are deleted, the migration task will also be deleted.

  1. In the migration task list, click a migration task name to go to the task details page.
  2. In the resource list, select the resources to be deleted and click Delete above the list.
  3. Confirm the selected resources, enter DELETE, and click Confirm.

    To prevent resource deletion failures caused by disconnection of the task from related services (such as SMS, OMS, and OBS), you can select Force Delete to forcibly delete the resources. However, this operation only deletes the resource data from MgC. It may not delete the related tasks and resources in the related cloud services like SMS, OMS, and OBS. You must delete any remaining resources manually.

Configuring Incremental Migration Settings

The following table describes the parameters about incremental migration.

Table 9 Configuring incremental migration settings

Parameter

Description

Overwrite Existing During Incremental Migration

See Table 6.

NOTE:

This setting only applies to incremental migration.

Incremental Migration Method
  • Manual: You can manually initiate incremental migration after the first incremental migration is complete.
  • Automated: You can define the interval, maximum attempts, and time zone to trigger incremental migrations.

    For example, if you configure the incremental migration settings as shown in the following figure, the system will initiate incremental migration at 08:00 in the morning on Mondays (GMT+08:00) for the next two weeks.
  • If you select Manual, incremental migration will be executed immediately when the workflow reaches the BatchCreateIncrementalMigrationTasks step.
  • If you select Automated, when the workflow reaches the BatchCreateIncrementalMigrationTasks step, you can:
    • Manually trigger incremental migration immediately when you are waiting for automatic incremental migration.
    • Skip incremental migration and verify services.
Parent Topic: Storage Migration