Help Center> Object Storage Service> Tools Guide> obsutil> Best Practices> Fine-Tuning obsutil Performance
View PDF

Fine-Tuning obsutil Performance

By default, obsutil uploads, downloads, and copies files or objects whose size is greater than 50 MB in multiple parts. Table 1 details related parameters in the .obsutilconfig file.

Table 1 Multipart-related parameters

Parameter

Description

defaultBigfileThreshold

Indicates the threshold for triggering multipart tasks, in bytes. If the size of a file to be uploaded, downloaded, or copied is greater than the threshold, the file is uploaded, downloaded, or copied in multiple parts. The default value is 50 MB.

defaultPartSize

Size of each part, in bytes. The default value is auto.

NOTE:
  • For multipart upload and copy, the value ranges from 100 KB to 5 GB.
  • For multipart download, the value is unrestricted.

defaultParallels

Maximum number of concurrent tasks in the multipart mode. The default value is 5.

Generally, multipart tasks not only speed up transmission but also allow you to resume failed tasks. By default, the part size of a multipart task can be automatically adjusted by the obsutil in the auto mode. In practice, however, to further improve the upload and download performance, you can adjust the part size according to the file size and the network conditions, to obtain the maximum transmission efficiency and ensure the successful completion of a transmission task.

Adjust the number of concurrent tasks in the multipart mode according to the following formula:

defaultParallels = Min(Number of CPUs x 2, Object size/defaultPartSize x 1.5)

In the upload, download, and copy commands, parameters -p and -ps are used to modify the number of concurrent tasks in the multipart mode and part size respectively, and then deliver the multipart task based on the parameter values configured in the command. The default values in the configuration file are used if you do not set them in a command.

Adjust the number of concurrent tasks in the multipart mode according to the following formula:

p = Min(Number of CPUs x 2, Object size/ps x 1.5)

  • Resources of a running host are limited. Therefore, if the number of concurrent tasks in the multipart mode is set too large, the performance of obsutil upload, download, or copy may deteriorate due to resource switchover and preemption between threads. In this case, you need to adjust the values of defaultParallels (-p) and defaultPartSize (-ps) based on the actual file size and network status. To perform a pressure test, lower the two values at first, and then gradually increase them to determine the optimal values.
  • If the values of defaultParallels (-p) and defaultPartSize (-ps) are too large, an EOF error may occur due to network instability. In this case, set the two parameters to smaller values.
  • If a batch operation is performed, the destination object size can be set to the average size of the objects to be operated.
Parent topic: Best Practices