Help Center/ Migration Center/ Best Practices/ Storage Migration/ Migrating Data from Other Cloud Platforms to Huawei Cloud
Updated on 2024-12-03 GMT+08:00
View PDF
Share

Migrating Data from Other Cloud Platforms to Huawei Cloud

Scenario

MgC allows you to quickly, easily migrate data to Huawei Cloud, from object storage to file storage, or from file storage to object storage. The choice is yours. This section describes how to use MgC to migrate data in object storage or file storage from other cloud platforms to Huawei Cloud.

The following regions are supported:

  • LA-Santiago
  • LA-Sao Paulo
  • TR-Istanbul
  • AP-Bangkok
  • AP-Singapore
  • AP-Jakarta
  • ME-Riyadh
  • CN North-Beijing4
  • CN East-Shanghai1

Highlights

  • Dedicated migration clusters reduce resource contention and improve migration efficiency.
  • Supported are object and file storage services on popular cloud platforms and self-built network storage systems. Supported migration sources include:
    • Huawei Cloud OBS
    • Alibaba Cloud OSS
    • Baidu Cloud BOS
    • Tencent Cloud COS
    • Kingsoft Cloud KS3
    • Qiniu Cloud KODO
    • UCloud US3
    • Amazon S3
    • Azure Blob Storage
    • NAS_GFS
    • NAS_SMB
    • NAS_NFS_V3_MOUNT
    • NAS_NFS_V3_PROTOCOL
    • HTTP/HTTPS data sources

Notes and Constraints

Architecture

The figure shows the logical architecture of storage migrations using MgC.

Figure 1 Logical architecture of storage migration

Preparations

Step 1: Create an OBS Bucket or SFS File System

MgC supports migrations from object storage to file storage and from file storage to object storage. Create an OBS bucket or create an SFS file system on Huawei Cloud as needed.

Step 2: Create a Migration Cluster

You can create dedicated migration clusters to accelerate storage migration. A migration cluster consists of nodes that have the migration plugin installed and are used to execute migration and listing tasks. For details, see Creating a Migration Cluster.

To migrate over a private line, map the private IP address used by the private line to the domain names of the target storage service and the target storage resource, such as a bucket, in the advanced settings.

Figure 2 Advanced settings

Step 3: Create a Storage Migration Workflow

  1. Sign in to the MgC console.
  2. In the navigation pane on the left, choose Migrate > Workflows. Select a migration project in the upper left corner of the page.
  3. Click Create Workflow in the upper right corner of the page.
  4. Select Storage Migration and click Configure Workflow. Click Preview Steps to view the migration stages and steps predefined in the template and the description of each stage and step. Steps of the Automated type will be automatically performed by MgC.
  5. Set workflow basics based on Table 1.

    Figure 3 Basic information
    Table 1 Basic parameters

    Parameter

    Description

    Name

    Enter a workflow name.

    Region

    Select a region you are migrating to. The migration cluster you want to use must be in this region.

    Description

    Enter a description.

    Cluster

    Select the migration cluster created in step 2.

  6. Configure the migration source and target based on Table 2 and Table 3.

    Figure 4 Source and target information
    Table 2 Parameters for configuring a migration source

    Parameter

    Description

    Remarks

    Location Type

    The supported migration sources include:

    • Huawei Cloud OBS
    • Alibaba Cloud OSS
    • Baidu Cloud BOS
    • Tencent Cloud COS
    • Kingsoft Cloud KS3
    • Qiniu Cloud KODO
    • UCloud US3
    • Amazon S3
    • Azure Blob Storage
    • NAS_GFS
    • NAS_SMB
    • NAS_NFS_V3_MOUNT
    • NAS_NFS_V3_PROTOCOL
    • HTTP/HTTPS data source

    -

    AK and SK

    Enter the AK/SK pair of the source cloud account. Ensure that the entered AK and SK are correct and you have the required permissions for the source bucket.

    These parameters are available when cloud storage is selected for Location Type.

    Bucket

    Enter the name of the source bucket to be migrated.

    Endpoint

    Enter the endpoint of the region where the source bucket is located.

    Type

    Set this parameter based on the source bucket type. You can view the bucket type in its basic information.

    This parameter is available when Huawei Cloud OBS is selected for Location Type.

    APPID

    Enter the APPID of your Tencent Cloud account.

    NOTE:

    You can view the APPID on the account information page of the Tencent Cloud console.

    This parameter is available when Tencent Cloud COS is selected for Location Type.

    Authentication Type

    The following authentication types are supported:

    • Connection string
    • Storage account name and key

    For details about how to obtain the credentials, see How Do I Obtain Credentials for Accessing Microsoft Azure?

    This parameter is available when Azure Blob Storage is selected for Location Type.

    List Path

    Enter the path where the lists of files to be migrated are stored. These lists must be stored in the same region as the target bucket.

    You need to write the URLs of files to be migrated and their new names at the target into the lists. Each line in the list can contain only one URL and one file name.

    Restrictions on list files are:

    • The files must be in .txt format, and their metadata Content-Type must be text/plain.
    • A single file can contain a maximum of 100,000 rows.
    • A single file cannot exceed 300 MB.
    • A maximum of 10,000 list files can be stored in the folder.
    • The files must be in UTF-8 without BOM.
    • The length of each line in a file cannot exceed 65,535 characters, or the migration will fail.
    • The Content-Encoding metadata of the files must be left empty, or the migration will fail.
    • In the files, a tab character (\t) must be used to separate the URL and new file name in each line. The format is [URL][Tab character][New file name]. Only the Chinese and special characters in the names must be URL encoded.
    • Spaces are not allowed in each line in a file. Spaces may cause migration failures because they may be mistakenly identified as object names.

    These parameters are available when HTTP/HTTPS data source is selected for Location Type.

    File System Address

    Enter the mount address of the source file system. The format is IP address:/xxx, for example, 192.1.1.1:/0001.

    These parameters are available when Location Type is set to NAS_SMB, NAS_NFS_V3_MOUNT, or NAS_NFS_V3_PROTOCOL.

    Path

    Enter the directory where files to be migrated are located. The format is /Folder name.

    Username

    Enter the username of the account that can access all files in the source file system, for example, administrator.

    These parameters are available when Location Type is set to NAS_SMB.

    Password

    Enter the password of the account.

    Domain on Windows

    Enter the domain of the target node.

    NOTE:

    You only need to enter the content before .com. For example, if the domain is test.com, enter test.
    Table 3 Parameters for configuring a migration target

    Parameter

    Description

    Remarks

    Location Type

    Select Huawei Cloud storage based on the source storage type.

    -

    AK

    Enter the AK of the Huawei Cloud account you are migrating to.

    These parameters are available when Location Type is set to Huawei Cloud OBS.

    SK

    Enter the SK of the Huawei Cloud account you are migrating to.

    Bucket

    Select the OBS bucket you are migrating your data to.

    Endpoint

    Enter the endpoint of the region where the target OBS bucket is located.

    NOTE:

    If the migration source is an OBS bucket, you can view the endpoint in the OBS bucket overview.

    Specify Prefix

    Specify a prefix to rename or relocate objects migrated to the target bucket. For example, if you specify the prefix /D, source file /A/B/C.txt will be relocated to /D/A/B/C.txt after being migrated to the target bucket. For details, see:

    Adding a Prefix or Directory Prefix to Migrated Objects

    File System Address

    Enter the mount address of the target file system. The format is IP address:/xxx, for example, 192.1.1.1:/0001.

    These parameters are available when Location Type is set to NAS_SMB or NAS_NFS_V3_MOUNT.

    Path

    Enter the directory for storing files migrated. The format is /Folder name.

    Username

    Enter the username of the account that can access all files in the target file system, for example, administrator.

    These parameters are available when Location Type is set to NAS_SMB.

    Password

    Enter the password of the account.

    Domain on Windows

    Enter the domain of the target node.

    NOTE:

    You only need to enter the content before .com. For example, if the domain is test.com, enter test.

  7. Configure the migration settings based on Table 4.

    Figure 5 Migration settings
    Table 4 Parameters for configuring the migration settings

    Parameter

    Value

    Description

    Task Type

    Full migration

    Migrates all data in the source bucket or specified paths.

    Partial migration by list

    Migrates files recorded in the list files.

    In List Path box, enter the path of the object lists stored in the target bucket. Restrictions on an object list file vary with the target location.

    • Target location: Huawei Cloud OBS
      • An object list file cannot exceed 30 MB.
      • An object list file must be a .txt file, and the Content-Type metadata must be text/plain.
      • An object list file must be in UTF-8 without BOM.
      • Each line in an object list file can contain only one object name, and the object name must be URL encoded.
      • Each line in an object list file cannot exceed 16 KB, or the migration will fail.
      • The Content-Encoding metadata of an object list file must be left empty, or the migration will fail.
      • An object list file can contain a maximum of 10,000 lines.
    • Target location: NAS
      • An object list file cannot exceed 30 MB.
      • An object list file must be a .txt file.
      • An object list file must be in UTF-8 without BOM.
      • Each line in an object list file can contain only one object name, and the object name must be URL encoded.
      • Each line in an object list file cannot exceed 16 KB, or the migration will fail.
      • An object list file can contain a maximum of 10,000 lines.

    Partial migration by prefix

    This option is only available for migration from cloud storage.

    If you enter a file name or name prefix in the Prefix text box, only the objects that exactly match the specified name or prefix are migrated.

    NOTICE:
    • If the files to be migrated are stored in the root directory of the source bucket, add their prefixes directly. If the files are stored in a non-root directory, add their directories and name prefixes in the format of Directory/Prefix.
    • Use commas (,) to separate multiple prefixes.

    Concurrent Subtasks

    -

    Specify the maximum number of concurrent subtasks. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if the number of online migration nodes is 2, the maximum number of subtasks can be 20 or any number below.

    Overwrite Existing

    Never

    Files existing at the migration target are never overwritten.

    WARNING:
    • If you choose Never for the initial migration, the attributes of involved parent folders at the source will not be migrated to the target. As a result, the folder attributes may be incomplete at the target. To avoid this issue, use the Never option with caution for the initial migration.
    • If a migration task is paused or interrupted and then restarted or resumed, the Never option will cause the system to skip files that were not completely migrated earlier, but the task may still be marked as successful. This affects data integrity. To avoid this issue, use the Never option with caution.

    Always

    Files existing at the migration target are always overwritten.

    If older or different size
    • Files that already exist at the target will be overwritten if they are older than or have different sizes from the paired files at the source.
    • Verification will be performed for folders after their contents are migrated. Folders that already exist at the target will be overwritten if they have different last modification times, sizes, or permissions from the paired folders at the source.
      NOTE:

      For empty folders, the overwrite policy is the same as that for files.

    Migrate Metadata

    -

    Determine whether to migrate metadata.

    • If you select this option, object metadata will be migrated.
    • If you do not select this option, only the Content-Type and Content-Encoding metadata will be migrated.

    Clear Cluster

    -

    Determine whether to clear the migration cluster after the migration is complete.

    • If you select this option, a step for clearing the migration cluster will be created in the workflow. You can also choose whether to clear resources used by the cluster, such as NAT gateways, security groups, and VPCEP resources.
    • If you do not select this option, a step for clearing the migration cluster will not be created in the workflow.

  8. (Optional) Configure advanced options based on Table 5.

    Figure 6 Advanced settings
    Table 5 Advanced options

    Parameter

    Description

    Remarks

    Target Storage Class

    Choose the storage class that your data will be migrated to in the target bucket. For details about storage classes, see Introduction to Storage Classes.

    -

    Enable KMS Encryption
    • If you do not select this option, whether migrated data will be encrypted in the target bucket depends on the server-side encryption setting of the bucket.
    • If you select this option, all migrated objects will be encrypted before they are stored in the target bucket.
    NOTE:
    • Using KMS to encrypt migrated data may slow down the migration speed by about 10%.
    • This option is only available when KMS is supported in the region you are migrating to.

    This parameter is only available for migrations to Huawei Cloud OBS.

    Restore Archive Data
    • If you do not select this option, the system records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
    • If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system skips it and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
    NOTE:

    The system will restore archive data before migrating it, and you pay the source cloud platform for the API requests and storage space generated accordingly.

    -

    Filter Source Data

    Filter files to be migrated using filters. For details about the filters, see Source Data Filters.

    Obtain Data from CDN

    If the default domain name cannot meet your migration requirements, then as long as the source cloud service provider supports custom domain names, you can associate a custom domain name to the source bucket, and enable the CDN service on the source platform to reduce data download expenses. Enter a custom domain name in the Domain Name text box and select a transmission protocol. HTTPS is more secure than HTTP and is recommended.

    If the migration source is the Alibaba Cloud OSS or Tencent Cloud COS, you also need to select an authentication type and enter an authentication key.

    Send SMN Notification

    Determine whether to use SMN to get notifications about migration results.

    • If you do not select this option, no SMN messages are sent after the migration.
    • If you select this option, after the migration, SMN messages are sent to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.

    Limit Traffic

    Allocate the maximum bandwidth to be used by the workflow during a specified period.

    • If you do not select this option, migration traffic is not limited.
    • If you select this option, limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit.
      For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s from 08:00 to 12:00. The migration speed is not limited beyond this period.
      NOTE:
      • The rate limit ranges from 0 MB/s to 1,048,576 MB/s.
      • A maximum of five rules can be added.
      • The time is the local standard time of the region you are migrating to.

    Schedule Migration

    Schedule the migration to automatically run during a period.

    • If you do not select this option, you need to manually start or stop the migration.
    • If you select this option, the migration runs during the specified period and stops beyond that period.

      For example:

      • If you set Start Time to 08:00 and End Time to 12:00, the migration task runs from 08:00 to 12:00 every day. The migration stops beyond that period.
      • If you set Start Time to 12:00 and End Time to 08:00, the migration runs from 12:00 of the current day to 08:00 of the next day. The migration stops beyond that period.

  9. Click Next: Confirm.
  10. Confirm the workflow settings and click Confirm. The Run Workflow dialog box is displayed, which indicates that the workflow has been created.

    • If you want to start the migration immediately, click Confirm to run the workflow.
    • If you want to add a stage or step to the workflow, click Cancel. The workflow enters a Waiting state, and the migration has not started. To start the migration, click Run in the Operation column.

  11. On the migration workflow details page, view the workflow settings and the migration progress. You can also perform the following operations:

    • Move the cursor to the migration progress bar of a resource. In the displayed window, view the migration details about the resource.
    • When a migration reaches a step that requires manual confirmation, place the cursor on the progress bar and click Confirm next to the step status in the displayed window. The migration can continue only after you confirm.
    • In the Basic Information area, click Manage next to the cluster name. The cluster details page is displayed on the right. On the displayed page, you can:
      • Add, edit, or delete traffic limiting rules to control cluster traffic based on your requirements.
      • Add or delete migration nodes or list nodes, or upgrade plug-ins for existing nodes as required.

  12. (Optional) Click the migration progress bar of a resource or click Migration Progress in the window displayed when you move course to the progress bar. The migration details page is displayed on the right. You can view the task overview and progress details. You can also perform the following operations:

    Operation

    Description

    Changing the migration cluster

    You can change the migration cluster only when the migration task (workflow) is Paused.

    1. In the Overview area, click Replace next to the migration cluster name.
    2. In the displayed dialog box, select a new cluster from the drop-down list and click Confirm. After the cluster is changed, the workflow starts to run automatically.

    Modifying the migration schedule
    1. In the Overview area, click Modify next to Schedule Migration.
    2. Set Start Time and End Time, and click Confirm.

    Modifying the number of concurrent subtasks
    1. In the Progress area, under Expected Concurrent Subtasks, click Modify to change the expected number of concurrent subtasks. There cannot be more than 10 concurrent subtasks for each online migration node. For example, if there are 2 online migration nodes, the maximum number of subtasks can be 20 or any number below.
    2. Click Confirm.

    Adding traffic limiting rules
    1. In the Migration Speed area, click Add to add a rule to limit the bandwidth the migration can use in a specified period.
      NOTICE:
      • The bandwidth limit ranges from 1 MB to 1,024 GB.
      • Time periods in different rules cannot overlap.

        For example, if there is a rule added for the period from 8:00 to 12:00, you cannot configure rules for any overlapped periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00.

      • The start time of a rule cannot be later than the end time.

        For example, the time period from 23:00 to 01:00 is not allowed.

    2. Click Save.

    Obtaining the lists of files that fail to be migrated, skipped, or migrated

    In the File Statistics area, view the path of the list of files that failed to be migrated, skipped, or migrated. Click a file path, and it will take you to the OBS bucket where the list is stored. You can download the list from the bucket.

    Viewing traffic statistics

    In the Traffic Statistics area, view the migration traffic in the last hour, last 6 hours, last 24 hours, or the entire migration period.

Source Data Filters

The following table describes the rules and restrictions for setting source data filters.

Table 6 Filter options

Option

Description

Patten Rule

Constraint

Exclude Patterns

If a file matches any excluded pattern, the file will not be migrated or compared for consistency. Both exact match and fuzzy match are supported.
  • Exact match
    You need to specify absolute paths and use slashes (\) to escape special characters in the paths.
    CAUTION:

    Precautions for configuring exclude and include patterns:

    • If the file system address ends with :/, when you configure the paths to be excluded or included, enter their absolute paths relative to the mount point.

      For example, if the file system address is 192.1.1.1:/ and the mount point is /mnt/turbo, enter absolute paths relative to /mnt/turbo.

      For example: 
      [root@oms-cluster-ecs filter_test]# pwd
/mnt/sts_turbo/mgc/filter_test
[root@oms-cluster-ecs- filter_test]# ll
drwxr-xr-x 2 root root 0 Aug 16 15:27 test2
-rw-r--r-- 1 root root 5 Aug 16 15:27 test2.log

      To exclude the test2.log file from the migration, you can enter its absolute path /mgc/filter_test/test2.log in the Exclude Patterns box.

    • If the file system address contains additional content after :/, when you configure paths to be excluded or included, combine the part after :/ in file system address with the absolute paths relative to the mount point.

      For example, if the file system address is 192.1.1.1:/mgc-test and the mount point is /mnt/turbo, combine /mgc-test with absolute paths relative to /mnt/turbo.

      For example: 
      [root@oms-cluster-ecs execution-service]# cd /mnt/turbo/autotest/filter_test/
[root@oms-cluster-ecs- filter_test]# ll
-rw-r--r-- 1 root root   14 Aug  8 09:22 test1.log
drwxr-xr-x 1 root root 4096 Aug  8 09:22 test2
-rw-r--r-- 1 root root   14 Aug  8 09:22 test2.log

      To exclude the test2.log file from the migration, enter /mgc-test/autotest/filter_test/test2 in the Exclude Patterns box.

  • Fuzzy match
    • An asterisk (*) matches zero or more characters except for slashes (/).
    • A pair of asterisks (**) matches zero or more characters including slashes (/).
    • A question mark (?) matches exactly one character, but not slashes (/).
    • Commas (,) are used to separate patterns in {}. Patterns in {} are in an OR relationship.
    • Wildcard characters asterisk (*) and question mark (?) are escaped by backslashes (\). In other cases, a backslash (\) means itself.
  • Except for {}, consecutive characters specified in pattern rules are not allowed, for example, ***, *?, **?, ?*, ?**, *{*, *}*, *}?, ?{*, {*}, {,}, {*,,, *}, and ,*,
  • Only asterisks (*) can be used as wildcard characters in {}.
  • {1} cannot be nested in {0}.
  • Excluded patterns take precedence over included patterns.
  • Semicolons (;) are used to separate patterns outside {}.

Include Patterns
  • If no included patterns are specified, all files in the source will be migrated.
  • If included patterns are specified, only the files whose absolute paths match the specified patterns will be migrated or compared for consistency.

Time Range

Filters files and directories to be migrated based on when they were last modified. Only files and directories whose last modification times fall in the configured time range will be migrated.

The start time and end time can be left empty. If they are left empty, the system will not filter out source files by time. The time can be precise to the minute.

The following table lists example pattern rules for different scenarios.

Assume that you want to migrate the directory test in the source storage system.

  • If the source storage system is a NAS device, enter /test as an "include" pattern.
  • If the source storage system is an object storage system, enter test as an "include" pattern.

Scenario

Example Pattern for NAS

Example Pattern for Object Storage

Description

File paths that end with xx

/xx /**xx

xx/**xx

xx can be an expression containing asterisks (*) and question marks (?).

Files whose names start with xx in the root directory

/xx*

  • /testssss matches the pattern.
  • /test/xx does not matches the pattern.

/xx*

  • testssss matches the pattern.
  • /test/xx does not matches the pattern.

File paths that start with xx

/xx**

/xx/**

xx**

xx/**

Files whose names contain xx

**xx*

**xx*

File paths that contain xx

**xx**

**xx**

File paths that start with xx and end with yy.

/xx**yy

xx**yy

xx and yy can be expressions containing asterisks (*) and question marks (?).

File paths that end with xx or yy

**{xx,yy}

**{xx,yy}

Files whose names contain xx or yy

**{xx,yy}*

**{xx,yy}*

Files paths that contain xx or yy

**{xx,yy}**

**{xx,yy}**
Parent topic: Storage Migration