Migrating Object Storage Data Using a List

Scenarios

List migration applies to migration of over 100,000 objects from Alibaba Cloud Object Storage Service (OSS) to Huawei Cloud Object Storage Service (OBS).

Prerequisites

• You have signed up for a HUAWEI ID and enabled Huawei Cloud services, and completed real-name authentication.
• You have obtained the AK/SK pairs for the source and target platform accounts.
  • The source platform account needs the permissions required to list all buckets, obtain bucket locations, list objects, obtain object metadata, and obtain object content. For details about how to obtain these permissions, see Source Permissions.
  • The target platform account needs the permissions required to list buckets, obtain bucket locations, list objects, obtain object metadata, obtain object content, upload objects, list uploaded parts, and restore archived objects. For details about how to obtain these permissions, see Target Permissions.
• You have buckets in OBS.

Procedure

1. Generate inventory lists for the Alibaba Cloud OSS bucket to be migrated.
   1. Sign in to the Alibaba Cloud OSS console and choose Buckets.
   2. On the Buckets page, click the name of the desired bucket.
   3. On the displayed page, choose Data Management > Bucket Inventory.
   4. On the Bucket Inventory page, click Create Inventory, enter a rule name, select an inventory storage bucket, and click OK.
   5. Return to the Bucket Inventory page and click View Inventory Lists in the Operation column. If a package exists in the data/ directory, the bucket inventory lists have been generated.

      

2. Process the bucket inventory lists.
   1. Prepare a Linux VM with 4 vCPUs, 8 GB memory, and 100 GB disk space (depending on the data volume) on Alibaba Cloud. Bind an EIP to the VM.
   2. Log in to the VM as root.
   3. Run the following command to install zip:

      yum install zip unzip

   4. Run the following command to install ossutil:

      sudo -v ; curl https://gosspublic.alicdn.com/ossutil/install.sh | sudo bash

   5. Configure ossutil.

      ossutil config
      Set the configuration file path as prompted.
        (1) Configuration file path
        (2) The display language (EN/CH)
        (3) AK/SK

   6. Download the bucket inventory lists.

      ossutil64 cp oss://<bucket-name>/<path>/data/ <CSV-file>/ --include "*.gz" -r
      # <bucket-name> indicates the name of the inventory storage bucket.
      # <path> indicates the path that stores the inventory lists.
       # <CSV-file> indicates the directory for storing downloaded CSV files
       # --include "*.gz" indicates that files whose names contain .gz are downloaded.
       # -r indicates that all eligible files within recursive directories are downloaded.

   7. Run the following command to install gzip:

      yum install gzip

   8. Run the following command to go to the directory where the package is stored:

      cd <CSV-file>
      # <CSV-file> indicates the directory for storing the files downloaded in step 2.5.

   9. Run the following command to decompress the bucket inventory file package:

      gzip -d *.csv.gz

   10. Select the corresponding CSV file and run the following command to extract object keys (URLs):

       cat <CSV-name> | awk -F ",\"" '{print $2}' | awk -F "\"" '{print $1}' > <List-file>
       # <CSV-name> indicates the name of the file to be processed.
       # <List-file> indicates the TXT file where the extracted data is saved.

   11. Split the list file. In the command below, xxxx.txt is the list file obtained in the previous step.

       #!/bin/bash
        sourceFile="xxxx.txt"
        splitLine=100000
        split -l $splitLine -d -a 5 --additional-suffix=.txt "$sourceFile"  xxxx-
        mkdir xxxx-file
        mv xxxx-*.txt xxxx-file

3. Save the lists of objects to be migrated generated in 2.k to the target bucket.
   1. Install obsutil on the VM prepared in 2.a. For details, see Downloading and Installing obsutil.
   2. Initialize the configuration.
   3. Use obsutil to upload the lists.

      ./obsutil cp <file-path> obs:// <bucket-name> -f -r
      # <file-path> indicates the folder where the list files are stored.
      # <bucket-name> indicates the name of the target bucket.

4. Create a storage migration workflow on MgC. Remember to use the list migration method.