Help Center/ Scalable File Service/ Best Practices/ Migrating Data of SFS Capacity-Oriented File Systems/ Migrating Data from SFS Capacity-Oriented to General-Purpose File System Using fpart
Updated on 2025-07-25 GMT+08:00
View PDF
Share

Migrating Data from SFS Capacity-Oriented to General-Purpose File System Using fpart

Solution Overview

This section describes how to use fpart to migrate data from SFS Capacity-Oriented to General-Purpose File System. The migration aims to efficiently and securely migrate data while ensuring data integrity and consistency. The entire process includes a full migration (no impact on services), a first incremental migration (no impact on services), unmounting the SFS Capacity-Oriented file system (services are interrupted), a second incremental migration (services are interrupted), and changing the mount point (services are interrupted).

Constraints

  • This solution is only applicable when a Linux ECS is used to migrate data.
  • SFS Capacity-Oriented file systems using CIFS or those mounted to Windows clients cannot be migrated.
  • The Linux ECS, SFS Capacity-Oriented file system, and general-purpose file system must be in the same VPC.
  • If the general-purpose file system is used by a CCE cluster, the CCE cluster must be upgraded to version 1.19.10, and the everest add-on version must be later than 2.0.9.

Solution Architecture

Figure 1 Process of migrating data from SFS Capacity-Oriented to General-Purpose File System using fpart

Resource Planning

Table 1 describes the resource planning in this solution.

Table 1 Resource planning

Resource

Example Configuration

Description

ECS

Specifications: 8 vCPUs | 16 GiB | c7.2xlarge.2

OS: Linux

Region: CN-Hong Kong

VPC: VPC1

Ensure that the /mnt/src and /mnt/dst directories have been created.

Preparations

  1. Create a general-purpose file system on the Huawei Cloud console in the VPC where the SFS Capacity-Oriented file system resides. If no VPC is available, create one in the same region and then create the general-purpose file system. For how to create a general-purpose file system, see Creating a General-Purpose File System. For how to create a VPC, see Creating a VPC with a Subnet.
  2. Buy a VPC endpoint (used for free) in the desired VPC. For how to buy a VPC endpoint, see Configure a VPC Endpoint.
    The following table shows the available VPC endpoint service names.
    Table 2 VPC endpoint service names

    Region

    Service Name

    CN North-Beijing4

    cn-north-4.com.myhuaweicloud.v4.storage.lz13

    CN South-Guangzhou (AZ6)

    cn-south-1.com.myhuaweicloud.v4.obsv2.storage.lz06

    CN East-Shanghai1

    cn-east-3.com.myhuaweicloud.v4.storage.lz07

    CN-Hong Kong

    ap-southeast-1.com.myhuaweicloud.v4.obsv2.storage.lz005
  3. If a Linux ECS is available, skip this step. If not, buy a Linux ECS on the Huawei Cloud console. You are advised to buy an ECS with 8 or 16 vCPUs. This ECS must be in the VPC where the SFS Capacity-Oriented file system and general-purpose file system belong and can communicate with the file systems.
  4. Install fpart on the Linux ECS. 
    yum -y install fpart rsync
  5. Create two local mount points on the Linux ECS. 
    mkdir /mnt/src
    mkdir /mnt/dst
  6. Obtain the mount points of the SFS Capacity-Oriented file system and general-purpose file system from the console. Then, mount the file systems on the created local mount points (/mnt/src and /mnt/dst). 
    mount -t nfs -o vers=3,timeo=600,nolock  <mount-point-of-the-SFS-Capacity-Oriented-file-system> /mnt/src
    mount -t nfs -o vers=3,timeo=600,nolock  <mount-point-of-the-general-purpose-file-system> mnt/dst
  7. If the mount fails, install the corresponding tool package by referring to Mounting an NFS File System to ECSs (Linux).

Procedure

  1. Full migration: Migrate all files from the SFS Capacity-Oriented file system to the general-purpose file system.

    1. Log in to the ECS using VNC and run the following command to synchronize data: 
      fpsync -n 500 -f 500 -o "-lptgoDvu --numeric-ids" -v /mnt/src/ /mnt/dst/
      • n: the number of parallel copy threads
      • f: the number of files each thread can copy
      • o: the rsync configuration option
      • v: the output detailed logs
      • If there are a large number of files, you can run the command concurrently for the subdirectories in /mnt/src.
      • You can adjust the n and f options based on the file and directory structure.
    2. The default run log files are stored in /tmp/fpsync, under which log stores task run logs, parts stores file partition logs, queue stores execution configurations, and work stores the copy commands. A run ID is returned after you run the fpsync command. You can use this run ID to check the task running status.

  2. First incremental migration: If there is data written to the source files from the service application during the full migration, synchronize the incremental data after the full migration is complete.

    Run the following command to perform an incremental migration: 
    fpsync -n 500 -f 500 -o "-lptgoDvu --numeric-ids" -v /mnt/src/ /mnt/dst/

    Take note of the start time and end time of this operation. This period is approximately equal to how long customer services need to be stopped in the following steps.

  3. Unmounting the SFS Capacity-Oriented file system: To prevent new data from being written to the file system, before synchronizing the incremental data, stop the service application that uses the source file system on all ECSs and containers.

    • Ensure that the SFS Capacity-Oriented file system is unmounted from all service ECSs (or cloud containers). Otherwise, after the migration, data in the general-purpose file system may be inconsistent with that in the SFS Capacity-Oriented file system.
    • Do not unmount the SFS Capacity-Oriented file system from the ECS that you use to perform the migration.
    • The operations vary in ECS and CCE scenarios. Refer to the operations based on your scenario.
    • Ensure that services are stopped when you perform this step.

    ECS scenario:

    1. Log in to each service ECS that mounts the SFS Capacity-Oriented file system.
    2. Unmount the SFS Capacity-Oriented file system. 
      umount -f <local-mount-point>

    CCE scenario:

    1. Log in to the CCE console, find the workload that uses the storage volume, and reduce the number of pods to 0.

    2. Wait until all pods are deleted and then go to the next step.

  4. Second incremental migration: Synchronize the incremental data generated during the first incremental migration to the general-purpose file system. After the synchronization, data in the general-purpose file system will be the same as that in the SFS Capacity-Oriented file system. 

    fpsync -n 500 -f 500 -o "-lptgoDvu --numeric-ids" -v /mnt/src/ /mnt/dst/
    • Check that data in the general-purpose file system is the same as that in the SFS Capacity-Oriented file system.
    • Ensure that services are stopped when you perform this step.

  5. Mounting the general-purpose file system: Unmount the SFS Capacity-Oriented file system from the ECS that you use to perform the migration and re-mount the general-purpose file system to this ECS.

    ECS scenario:

    1. Log in to the ECS that you use to perform the migration and run the following command to unmount the SFS Capacity-Oriented file system: 
      umount -f <local-mount-point>
    2. Mount the general-purpose file system. 
      mount -t nfs -o vers=3,timeo=600,noresvport,nolock <mount-point-of-the-general-purpose-file-system>  <local-mount-point>

    CCE scenario: Mount the file system by referring to Migrating Containerized Application Data from SFS 1.0 to General-Purpose File System or SFS Turbo.

    • Ensure that the mount parameters and directories are correct. Or, services may be affected.
    • The operations vary in ECS and CCE scenarios. Refer to the operations based on your scenario.

  6. The customer restores services and checks whether services are normal.
  7. After observing the services for a period of time, the customer can release the SFS Capacity-Oriented file system resources to prevent unintended charges.