Help Center/ Object Storage Migration Service/ User Guide/ Synchronization Tasks/ Creating a Synchronization Task
Updated on 2024-05-09 GMT+08:00
View PDF
Share

Creating a Synchronization Task

Scenario

A synchronization task allows you to proactively synchronize new or modified objects from the source bucket to the destination bucket. You need to configure an event trigger for the source object storage service. When a source object is added or modified, the source object storage service calls the synchronization API of OMS through a function workflow, and the API will synchronize the object to the destination OBS bucket on Huawei Cloud.

Synchronization tasks depend on source event triggers, function workflows, and public network transmission, which may be unstable and cannot ensure 100% synchronization success. Therefore, you are not advised to use this feature for data synchronization in DR scenarios.

Scenarios

Enter the AKs and SKs for accessing the source and destination buckets, and configure the synchronization task parameters for migrating source objects contained in the synchronization request and viewing the task details.

The synchronization task feature is available only in CN North-Beijing4 and CN East-Shanghai1.

Prerequisites

  • You have obtained the AK/SK pairs and required permissions for the source and destination platform accounts.
    • The source platform account needs the permissions for:
      • Listing buckets
      • Obtaining bucket locations
      • Listing objects
      • Obtaining object metadata
      • Obtaining object content

      For details about how to obtain these permissions, see Source Permissions.

    • The destination platform account needs the permissions for:
      • Listing buckets
      • Obtaining bucket locations
      • Listing objects
      • Obtaining object metadata
      • Obtaining object content
      • Listing uploaded parts
      • Restoring archive objects

      For details about how to obtain these permissions, see Destination Permissions.

  • You have created at least one bucket on OBS.
  • You have created less than 1,000,000 migration tasks within the past 24 hours.
  • You have less than 1,000,000 migration tasks in Waiting state.
  • You have less than 5 synchronization tasks in the Synchronizing state.

Procedure

  1. Log in to the management console and go to the Object Storage Migration Service page.
  2. In the navigation pane on the left, choose Synchronization Tasks.
  3. In the upper right corner, click Create Synchronization Task.
  4. Read the service disclaimer, select I have read and agree to the service disclaimer, and click OK. The page for creating a synchronization task is displayed.
  5. For step Select Source/Destination, you need to:

    1. Set the source parameters by referring to Table 1.
    Table 1 Source parameters

    Parameter

    Description

    Data Source

    Specifies the source cloud service provider. The following options are available:

    • Huawei Cloud
    • AWS
    • Alibaba Cloud
    • Microsoft Azure
    • Tencent Cloud
    • Google Cloud
    • Qiniu Cloud
    • QingCloud
    • Baidu Cloud
    • Kingsoft Cloud
    • UCloud

    appID

    Specifies the identifier of the account for logging in to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, appID must be specified.

    Access Key ID

    Specifies the AK for accessing the source platform. It can contain a maximum of 100 characters.

    Secret Access Key

    Specifies the SK of the source platform account. The SK matches the AK, and they group into a pair. It can contain a maximum of 100 characters.

    Specify Source Bucket

    Enable this option if your source account does not have the permissions required to list buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects. Then specify the bucket or path to be migrated.

    For details, see Table 2.
    Table 2 Description of Specify Source Bucket

    If You Need to...

    Then...

    Enable Specify Source Bucket.
    1. Select the region where the source bucket resides from the Region drop-down list.
    2. In the Bucket text box, enter the bucket or the path to be migrated. The path must have the bucket name included.
    3. Click Connect.

      OMS will check whether the source platform can be connected using the information you entered. If the source platform can be connected, the system will prompt you.

    Disable Specify Source Bucket (This feature is disabled by default.)
    1. Click List Buckets.

      OMS will check whether the source platform can be connected using the information you entered. After the source platform is connected, all buckets in your account will be listed in the Bucket drop-down list on the premises that your account has the permissions needed to list buckets.

    2. In the Bucket drop-down list, select the bucket where the data to be migrated is stored.
    2. Set destination parameters by referring to Table 3.
    Table 3 Destination parameters

    Parameter

    Description

    Access Key ID

    Enter the AK for accessing Huawei Cloud. It can contain a maximum of 100 characters.

    Secret Access Key

    Enter the SK for accessing Huawei Cloud. The SK matches the AK, and they group into a pair. It can contain a maximum of 100 characters.

    After entering the AK and SK, click Connect to Destination Bucket.

    Bucket

    Select a destination OBS bucket for storing the migrated data.

  6. Click Next to set task parameters. For details about the parameters, see Table 4.

    Table 4 Task parameters

    Parameter

    Description

    Object Metadata
    • If you select Do not migrate, OMS will only migrate the metadata ContentType.
    • If you select Migrate, OMS will migrate supported metadata.

    Data Encryption

    If you select KMS, migrated data is encrypted before being transferred to the destination OBS buckets.

    NOTE:

    Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

    This option is only available if KMS encryption is supported in the current region.

    Archive Data Restoration

    Archive data must be restored before being migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

    • If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
      CAUTION:
      • Currently, the following cloud platforms can automatically restore archive objects: Huawei Cloud, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.
      • The system will restore archive data before migrating it, and you will pay to the source cloud service provider for the API requests and storage space generated accordingly.
      • The copies generated for restoring archive data will be stored for three days by default. If there is more than 3 TB of data or more than 3 million objects in a source bucket, create a migration task group to migrate data in case the migration fails because the restored data is archived again.
      • Pausing or restarting a migration task will result in restoration of all archive data again, no matter it has been restored or not.
      • After data is migrated from the source platform to Huawei Cloud, the storage class of the migrated data is the same as that of the destination bucket.

      If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated, skips them, and continues to migrate other objects in the migration task.

      NOTE:
      • Currently, the following cloud service providers can automatically restore archive objects: Huawei Cloud, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.
      • It takes at least a minute to restore an archive object. The larger the archive object is, the longer it takes to restore it, but if an archive object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see the documentation of the specific cloud service provider.
      • Two types of fees are generated during archive data restoration: fees for API calls and fees for archive object restoration. These fees are billed by the source cloud service provider.
      • After the restoration is complete, archive objects will be in restored state for a certain period of time. During this period, objects can be downloaded or read. After this period expires, archive objects need to be restored again for migration. For details about how long they stay in the restored state, see the documentation of the cloud service provider.
      • After objects are migrated from the source platform to Huawei Cloud, they are in the same storage class as the destination bucket.

    Consistency Check

    By default, object consistency is checked by size and last modification time.

  7. Click Next to enter the confirmation page.

    Confirm the settings of the synchronization task and start the synchronization.

    1. Check whether the settings of synchronization parameters are correct.

    If the settings are incorrect, click Previous to return to the previous page for modification.

    2. Click Synchronize Now.

    A message is displayed indicating that the task has been submitted. Switch back to the synchronization task list to view the task.

    For details about task statuses, see Table 5.

    Table 5 Synchronization task statuses

    Status

    Description

    Synchronizing

    Indicates that the objects in the source bucket are being synchronized.

    NOTE:

    After you create a synchronization task, the task status is Synchronizing, which indicates means that the synchronization request from the source bucket can be received and the source objects can be synchronized. You need to configure a synchronization request on the source platform to migrate newly added or modified objects in the source bucket. For more details, see Configuring the Source Storage Service to Send Synchronization Requests. After the configuration, OMS will synchronize your source objects.

    Paused

    Indicates that the synchronization of source objects is paused.

    NOTE:

    If you want to pause a synchronization task, you are advised to stop writing data to the source bucket and wait for 15 minutes before pausing the synchronization task. Otherwise, it may cause a migration failure for some objects covered by the received synchronization requests.

  8. Start, pause, or delete the synchronization task.

    • Starting the synchronization task
    • Only synchronization tasks in the Paused state can be started.
    • For security purposes, the system does not save your personal data. When you restart the synchronization task, you need to enter the AKs and SKs for accessing the source and destination buckets again to complete identity authentication.
    • Pausing the synchronization task
    • Only synchronization tasks in the Synchronizing state can be paused.
    • If you want to pause a synchronization task, you are advised to stop writing data to the source bucket and wait for 15 minutes before clicking Pause on the synchronization task page. Otherwise, it may cause a migration failure for some objects covered by the received synchronization requests.
    • Deleting the synchronization task
    • Only synchronization tasks in the Paused state can be deleted.

Parent topic: Synchronization Tasks