Migrating Data to Huawei Cloud

Scenario

MgC allows you to quickly, easily migrate data to Huawei Cloud, from object storage to file storage, or from file storage to object storage. The choice is yours. This section describes how to use MgC to migrate data in object storage or file storage from other cloud platforms to Huawei Cloud.

The following regions are supported:

• CN North-Beijing4
• CN South-Guangzhou
• CN East-Shanghai1
• CN South-Guangzhou-InvitationOnly

Highlights

• Dedicated migration clusters reduce resource contention and improve migration efficiency.
• Supported are object and file storage services on popular cloud platforms and self-built network storage systems. Supported migration sources include:
  • Huawei Cloud OBS
  • Alibaba Cloud OSS
  • Baidu Cloud BOS
  • Tencent Cloud COS
  • Kingsoft Cloud KS3
  • Qiniu Cloud KODO
  • UCloud US3
  • Amazon S3
  • Azure Blob Storage
  • QingCloud QingStor
  • NAS_SMB
  • NAS_NFS_V3_MOUNT
  • NAS_NFS_V3_PROTOCOL
  • HTTP/HTTPS data sources

Architecture

The figure shows the logical architecture of storage migrations using MgC.

Figure 1 Logical architecture of storage migration

Preparations

• Prepare a Huawei account or an IAM user that can access MgC. For details, see Preparations.
• Create a migration project on the MgC console.
• Ensure that the source and target accounts have the permissions required for the migration. For details, see How Do I Obtain Required Permissions for the Source and Target Accounts?

Step 1: Create an OBS Bucket or SFS File System

MgC supports migrations from object storage to file storage and from file storage to object storage. Create an OBS bucket or create an SFS file system on Huawei Cloud as needed.

Step 2: Create a Storage Migration Workflow

1. Log in to the MgC console.
2. In the navigation pane on the left, choose Migrate > Workflows. Select a migration project in the upper left corner of the page.
3. Click Create Workflow in the upper right corner of the page.
4. Select Storage Migration and click Configure Workflow. Click Preview Steps to view the migration stages and steps predefined in the template and the description of each stage and step. Steps of the Automated type will be automatically performed by MgC.
5. Set workflow basics based on Table 1.

   Table 1 Basic parameters

   Parameter	Description
   Name	Enter a workflow name.
   Region	Select a region where you want to migrate to.
   Description	Enter a description.
   Cluster	Select a migration cluster. The cluster must contain migration nodes and execution nodes. If no cluster is available, create a cluster.

6. Configure the migration source and target based on Table 2 and Table 3.

   Table 2 Parameters for configuring a migration source

   Parameter	Description	Remarks
   Location Type	The supported migration sources include: Huawei Cloud OBS Alibaba Cloud OSS Baidu Cloud BOS Tencent Cloud COS Kingsoft Cloud KS3 Qiniu Cloud KODO UCloud US3 Amazon S3 Azure Blob Storage NAS_SMB NAS_NFS_V3_MOUNT NAS_NFS_V3_PROTOCOL HTTP/HTTPS data source	-
   AK	Enter the AK of the source cloud account.	These parameters are available when cloud storage is selected for Location Type.
   SK	Enter the SK of the source cloud account.
   Bucket	Enter the name of the source bucket to be migrated.
   Endpoint	Enter the endpoint of the region where the source bucket is located.
   Type	Set this parameter based on the source bucket type. You can view the bucket type in its basic information.	This parameter is available when Huawei Cloud OBS is selected for Location Type.
   APPID	Enter the APPID of your Tencent Cloud account. NOTE: You can view the APPID on the account information page of the Tencent Cloud console.	This parameter is available when Tencent Cloud COS is selected for Location Type.
   List Path	Enter the path for storing the lists that record the files to be migrated. These lists must be stored in the same region as the target bucket. You need to write the URLs of files to be migrated and their new names at the target into the lists. Each line in the list can contain only one URL and one file name. Restrictions on list files are: The files must be in .txt format, and their metadata Content-Type must be text/plain. A single file can contain a maximum of 100,000 rows. A single file cannot exceed 300 MB. A maximum of 10,000 list files can be stored in the folder. The files must be in UTF-8 without BOM. The length of each line in a file cannot exceed 65,535 characters, or the migration will fail. The Content-Encoding metadata of the files must be left empty, or the migration will fail. In the files, a tab character (\t) must be used to separate the URL and new file name in each line. The format is [URL][Tab character][New file name]. Only the Chinese and special characters in the names must be URL encoded. Spaces are not allowed in each line in a file. Spaces may cause migration failures because they may be mistakenly identified as object names.	These parameters are available when HTTP/HTTPS data source is selected for Location Type.
   File System Address	Enter the mount address of the source file system. The format is IP address:/xxx, for example, 192.1.1.1:/0001.	These parameters are available when Location Type is set to NAS_SMB, NAS_NFS_V3_MOUNT, or NAS_NFS_V3_PROTOCOL.
   Path	Enter the directory where files to be migrated are located. The format is /Folder name.
   Username	Enter the username of the account that can access all files in the source file system, for example, administrator.	These parameters are available when Location Type is set to NAS_SMB.
   Password	Enter the password of the account.
   Domain on Windows	Enter the domain that the account has joined in. NOTE: You only need to enter the content before .com. For example, if the domain is test.com, enter test.

   Table 3 Parameters for configuring a migration target

   Parameter	Description	Remarks
   Location Type	Select Huawei Cloud storage based on the source storage type.	-
   AK	Enter the AK of the Huawei Cloud account you are migrating to.	These parameters are available when Location Type is set to Huawei Cloud OBS.
   SK	Enter the SK of the Huawei Cloud account you are migrating to.
   Bucket	Select the OBS bucket you are migrating your data to.
   Endpoint	Enter the endpoint of the region where the target OBS bucket is located. NOTE: If the migration source is an OBS bucket, you can view the endpoint in the OBS bucket overview.
   Specify Prefix	Specify a prefix for files migrated to the target bucket. For example, if you specify a prefix /D, source file /A/B/C.txt will be migrated to /D/A/B/C.txt at the target. For details, see Adding a Prefix or Directory Prefix to Migrated Objects.
   File System Address	Enter the mount address of the target file system. The format is IP address:/xxx, for example, 192.1.1.1:/0001.	These parameters are available when Location Type is set to NAS_SMB or NAS_NFS_V3_MOUNT.
   Path	Enter the directory for storing files migrated. The format is /Folder name.
   Username	Enter the username of the account that can access all files in the target file system, for example, administrator.	These parameters are available when Location Type is set to NAS_SMB.
   Password	Enter the password of the account.
   Domain on Windows	Enter the domain that the account has joined in. NOTE: You only need to enter the content before .com. For example, if the domain is test.com, enter test.

7. Configure the migration task based on Table 4.

   Table 4 Parameters for configuring a migration task

   Parameter	Value	Description
   Task Type	Full migration	Migrates all data in the source bucket or specified paths.
   	Partial migration by list	Migrates files recorded in the list files.
   	Partial migration by prefix	This option is only available for migrations from cloud storage. If you enter a file name or name prefix in the Prefix text box, only the objects that exactly match the specified name or prefix are migrated. NOTE: If the files to be migrated are stored in the root directory of the source bucket, add their prefixes directly. If the files are stored in a non-root directory, add their directories and their prefixes in the format of Folder name/Prefix.
   Concurrent Subtasks	-	Specify the maximum number of concurrent subtasks. The value cannot exceed the number of online migration nodes multiplied by 5. For example, if the number of online migration nodes is 2, the maximum number of subtasks can be 10 or any number below.
   Overwrite Existing	Never	Files existing at the migration target are never overwritten.
   	Always	Files existing at the migration target are always overwritten.
   	If older or different size	Files existing at the migration target are overwritten if they are older than or have different sizes from files at the migration source.
   Migrate Metadata	-	Determine whether to migrate metadata. If you select this option, object metadata will be migrated. If you do not select this option, only the ContentType metadata will be migrated.
   Clear Cluster	-	Determine whether to clear the migration cluster after the migration is complete. If you select this option, a step for clearing the migration cluster will be created in the workflow. You can also choose whether to clear resources used by the cluster, such as NAT gateways, security groups, and VPCEP resources. If you do not select this option, a step for clearing the migration cluster will not be created in the workflow.

8. (Optional) Configure advanced options based on Table 5.

   Table 5 Advanced options

   Parameter	Description	Remarks
   Enable KMS Encryption	If you do not select this option, objects are in the same encryption status before and after the migration. If you select this option, all migrated objects will be encrypted before they are stored in the target bucket. NOTE: Using KMS to encrypt migrated data may decrease the migration speed by about 10%. This option is available only when KMS is supported in the region you are migrating to.	This parameter is only available for migrations to Huawei Cloud OBS.
   Restore Archive Data	Only restored data can be migrated. You can select this option if the source cloud platform supports automatic restoration of archive data. Currently, the following cloud platforms can automatically restore archive objects: Huawei Cloud, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud. If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system skips and records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. NOTE: The system will restore archive data before migrating it, and you will pay to the source cloud platform for the API requests and storage space generated accordingly.	-
   Filter Source Data	Filter files to be migrated using filters. For details about the filters, see Source Data Filters.
   Limit Traffic	Allocate the maximum bandwidth to be used by the workflow during a specified period. If you do not select this option, migration traffic is not limited. If you select this option, limit the migration traffic by setting Start Time, End Time, and Bandwidth Limit. For example, if you set Start Time to 08:00, End Time to 12:00, and Bandwidth Limit to 20 MB/s, the maximum migration speed is limited to 20 MB/s when the migration task is running from 08:00 to 12:00. The migration speed is not limited beyond this period. NOTE: The bandwidth limit ranges from 1 MB/s to 200 MB/s. You can create a maximum of 11 rules. The time is the local standard time of the region you are migrating to.
   Obtain Data from CDN	If the default domain name cannot meet your migration requirements, then if the source cloud service provider supports custom domain names, you can bind a custom domain name to the source bucket, and enable the CDN service on the source platform to reduce data download fees. Enter a custom domain name in the Domain Name text box and select a transmission protocol. HTTPS is more secure than HTTP and is recommended. If the migration source is the Alibaba Cloud OSS or Tencent Cloud COS, you also need to select an authentication type and enter an authentication key.
   Send SMN Notification	Determine whether to use SMN to notify you of migration results. If you do not select this option, no SMN messages are sent after the migration. If you select this option, SMN messages are sent after the migration to the subscribers of the selected topic. You can select the language and trigger conditions for sending messages.

9. Click Next: Confirm.
10. Confirm the workflow settings, and click Confirm. The Run Workflow dialog box is displayed, which indicates that the workflow has been created.
    • If you want to start the migration immediately, click Confirm to run the workflow.
    • If you want to add a stage or step to the workflow, click Cancel. The workflow goes to a Waiting state and the migration is not started. To start the migration, click Run in the Operation column.

11. On the migration workflow details page, view the workflow settings and the migration progress. You can also perform the following operations:
    • Move the cursor to the migration progress bar of a resource. In the displayed window, view the migration details about the resource.
    • When a migration reaches a step that requires manual confirmation, place the cursor on the progress bar and click Confirm next to the step status in the displayed window. The migration can continue only after you confirm.
    • In the Basic Information area, click Manage next to the cluster name. The cluster details page is displayed on the right. On the displayed page, you can:
      • Add, edit, or delete traffic limiting rules to control cluster traffic based on your requirements.
      • Add or delete migration nodes, list nodes, or upgrade plug-ins for existing nodes as required.

12. (Optional) Click the migration progress bar of a resource or click Migration Progress in the window displayed when you move course to the progress bar. The migration details page is displayed on the right. You can view the task overview and progress details. You can also perform the following operations:

    Operation	Description
    Modify the concurrent subtasks.	In the Progress area, click Modify under Expected Concurrent Subtasks to change the expected number of concurrent subtasks. The maximum number of concurrent subtasks cannot exceed the number of online migration nodes multiplied by 20. For example, if the number of online migration nodes is 2, the maximum number of concurrent subtasks is 40. Click Confirm.
    Add traffic limiting rules.	In the Migration Speed area, click Add to add a rule to limit the bandwidth the migration can use in a specified period. NOTICE: The bandwidth limit ranges from 1 MB to 1,024 GB. Time periods in different rules cannot overlap. For example, if there is a rule added for the period from 8:00 to 12:00, you cannot configure rules for any overlapped periods, such as from 7:00 to 13:00, 7:00 to 8:00, and 9:00 to 12:00. The start time cannot be later than the end time in a rule. For example, the time period from 23:00 to 01:00 is not allowed. Click Save.
    Obtain the list of files that fail to be migrated, skipped or migrated.	In the File Statistics area, view the path for storing the list of files that fail to be migrated, skipped, or migrated. Click a path, and you will navigate to the OBS bucket where the list is stored. You can download the list from the bucket.
    View traffic statistics.	In the Traffic Statistics area, view the migration traffic in the last hour, last 6 hours, last 24 hours, or the entire migration period.

Creating a Cluster

1. In the cluster drop-down list, click Create Cluster.
2. Set cluster parameters based on Table 6.

   Table 6 Parameters for creating a cluster

   Area	Parameter	Description
   Basic Configuration	Cluster Name	Enter a name.
   	VPC	Select a VPC from the drop-down list.
   	Subnet	Select a subnet. Make sure that the number of unused IP addresses in the subnet is at least 1 larger than the total number of migration nodes and list nodes in the cluster.
   Migration Node	ECS Specifications	The recommended specifications are 8 vCPUs and 16 GB of memory.
   	Quantity	The number of nodes must meet the following requirements: Number of migration nodes + Number of list nodes + 1 ≤ 100 Number of migration nodes + Number of list nodes + 1 ≤ Number of unused IP addresses in the subnet
   List Node	ECS Specifications	The recommended specifications are 8 vCPUs and 16 GB of memory.
   	Quantity	The number of nodes must meet the following requirements: Number of migration nodes + Number of list nodes + 1 ≤ 100 Number of migration nodes + Number of list nodes + 1 ≤ Number of unused IP addresses in the subnet
   Network	Public	Select a public NAT gateway. If no gateway is available, choose Buy Gateway from the drop-down list and select the gateway specifications and EIPs you want to associate with the gateway. A maximum of 20 EIPs can be selected at a time.
   	Private	Enter an IP address of such as Nginx or gateway that is allowed to forward or send requests over the private line.
   	Log Collection	If this option is enabled, logs generated during storage migrations are collected for locating problems if any. If this option is disabled, logs generated during storage migrations are not collected.

Source Data Filters

The following table describes the rules and restrictions for setting source data filters.