How Do I Configure the Maximum Number of Concurrent Migration Subtasks?

Background

In a migration task, the maximum number of concurrent subtasks (threads) directly affects the CPU usage, network throughput, and migration speed. Too few threads lead to resource underutilization, while too many threads cause excessive context switching, degrading overall performance.

Based on actual test data, the following section provides recommended concurrent subtask configurations for different file sizes to optimize migration efficiency.

Small Files (< 1 KB, such as 40 bytes)

• Processing such small files incurs high CPU computational overhead.
• The serialization and metadata processing of each object consume CPU resources.
• Excessive threads lead to severe thread context switching, which reduces the migration efficiency.

Small Files (< 1 MB, such as 30 KB)

• CPU consumption remains relatively high when such small files (< 1 MB, such as 30 KB) are processed.
• Compared with 40-byte files, the I/O ratio slightly increases.
• You can increase the number of concurrent tasks to maximize CPU utilization.

Medium Files (< 10 MB, such as 1 MB)

• The file size is moderate, and the CPU and I/O load is well balanced.
• Larger files result in more time spent waiting for I/O operations.
• More concurrent tasks are supported.

Large Files (≥ 10 MB)

• The time required for migrating large files (≥ 10 MB) is heavily dominated by I/O operations.
• During the migration of large files, threads spend most of their time waiting for network I/O operations.
• Higher concurrency is supported to fully utilize the wait time.

Quick Reference Table