Help Center/ Object Storage Service/ Tool Guide (obsutil) (Ankara Region)/ Parameter Description
Updated on 2024-04-17 GMT+08:00
View PDF
Share

Parameter Description

You can set obsutil parameters in the .obsutilconfig file.

Configuration file format:

endpoint=<value>
ak=<value>
sk=<value>
token=<value>
endpointCrr=<value>
akCrr=<value>
skCrr=<value>
tokenCrr=<value>
connectTimeout=<value>
socketTimeout=<value>
maxRetryCount=<value>
maxConnections=<value>
defaultBigfileThreshold=<value>
defaultPartSize=<value>
defaultParallels=<value>
defaultJobs=<value>
defaultJobsCacheCount=<value>
rateLimitThreshold=<value>
sdkLogBackups=<value>
sdkLogLevel=<value>
sdkLogPath=<value>
sdkMaxLogSize=<value>
utilLogBackups=<value>
utilLogLevel=<value>
utilLogPath=<value>
utilMaxLogSize=<value>
writeBufferIoSize=<value>
readBufferIoSize=<value>
recordMaxLogSize=<value>
recordBackups=<value>
humanReadableFormat=<value>
showProgressBar=<value>
showStartTime=<value>
helpLanguage=<value>
defaultTempFileDir=<value>
checkSourceChange=<value>
skipCheckEmptyFolder=<value>
fsyncForDownload=<value>
memoryEconomicalScanForUpload=<value>
forceOverwriteForDownload=<value>
panicForSymbolicLinkCircle=<value>
fastFailThreshold=<value>
abortHttpStatusForResumableTasks=<value>
showBytesForCopy=<value>
proxyUrl=<value>
faultTolerantMode=<value>

Table 1 describes the parameters.

Table 1 obsutil configuration parameters

Parameter

Optional or Mandatory

Description

Recommended Value

endpoint

Mandatory

Endpoint for accessing OBS, which can contain the protocol type, domain name, and port number (optional), for example, https://your-endpoint:80. For security purposes, you are advised to use HTTPS.

NOTE:
  • Obtain information about the regions and endpoints from the enterprise administrator.
  • If the configured endpoint does not contain any protocol, the HTTPS protocol is used by default.

N/A

ak

Mandatory

Access key ID.

NOTE:

N/A

sk

Mandatory

Secret access key.

NOTE:

N/A

token

Optional

Security token. If this parameter is empty, the security token is not set. When temporary AK and SK are used to access OBS, the token must be carried.

N/A

endpointCrr

Optional

Endpoint for accessing OBS in the region where the source bucket resides when the client-side cross-region replication function is enabled, which can contain the protocol type, domain name, and port number, for example, https://your-endpoint:80.

NOTE:
  • Obtain information about the regions and endpoints from the enterprise administrator.
  • If the configured endpoint does not contain any protocol, the HTTPS protocol is used by default.

N/A

akCrr

Optional

AK for the source bucket when the client-side cross-region replication function is enabled

N/A

skCrr

Optional

SK for the source bucket when the client-side cross-region replication function is enabled

N/A

tokenCrr

Optional

Security token used by the source bucket when the client-side cross-region replication is used. If this parameter is left blank, no security token is set. This parameter is mandatory when temporary access keys are used for access.

N/A

connectTimeout

Optional

Timeout interval for establishing an HTTP/HTTPS connection, in seconds. The default value is 30.

[5, 120]

socketTimeout

Optional

Timeout interval for reading and writing data, in seconds. The default value is 310.

[5, 600]

maxRetryCount

Optional

Maximum number of retry attempts. The default value is 3.

NOTE:

When an OBS request completes but HTTP status code 408 or 5XX is returned, or when a timeout error occurs in an OBS request, the request is retried.

[0, 5]

maxConnections

Optional

Maximum number of HTTP connections that can be accessed. The default value is 1000.

N/A

defaultBigfileThreshold

Optional

Threshold for triggering a multipart upload, in bytes. If the size of a file to be uploaded, downloaded, or copied is greater than the specified threshold, the file will be uploaded, downloaded, or copied using a multipart upload. The default value is 50MB.

NOTE:

This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

Greater than 5MB

defaultPartSize

Optional

Size of each part, in bytes. The default value is auto.

NOTE:
  • For a multipart upload and copy, the value ranges from 100KB to 5GB.
  • For multipart download, the value is unrestricted.
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).
  • If this parameter is set to auto. In this case, obsutil automatically sets the part size for each multipart task based on the source object size.

[9MB, 100MB]

defaultParallels

Optional

Maximum number of concurrent tasks in the multipart mode. The default value is 5.

Set this parameter according to Fine-Tuning obsutil Performance.

defaultJobs

Optional

Maximum number of concurrent tasks in batches. The default value is 5.

NOTE:

Batch tasks include uploading, downloading, and copying folders, as well as deleting objects in batches.

[1, 50]

defaultJobsCacheCount

Optional

Cache size of a batch task queue, indicating the maximum number of tasks that can be cached. The default value is 1000000.

NOTE:

More cached tasks consume more memory resources. Therefore, you are advised to adjust the value of this parameter based on site requirements.

Default value

rateLimitThreshold

Optional

Traffic control threshold of an upload or a download request, in bytes per second. The default value is 0, indicating that traffic is not limited. The minimum value is 10KB.

NOTE:

This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

Greater than 100KB

sdkLogBackups

Optional

Maximum number of SDK log files that can be retained. The default value is 10.

N/A

sdkLogLevel

Optional

SDK log level. Possible values are:

  • DEBUG
  • INFO
  • WARN
  • ERROR

The default value is WARN.

N/A

sdkLogPath

Optional

Absolute path of SDK logs. The value must be a file path. The default value is the path of the obssdk.log file in the subfolder obsutil_log of the user's home directory (HOME in Linux or macOS and C:\Users\<Username> in Windows).

NOTE:
  • If this parameter is left blank, no SDK log is generated.
  • The path must be a file path and cannot be a folder path.
  • After the SDK log function is enabled, all logs of requests to OBS are saved in the SDK log file for problem analysis and location.
  • Ensure that the user who runs the command has the read and write permissions on the path.
NOTICE:

If multiple obsutil processes are running at the same time, log files may fail to be written concurrently or may be lost. In this case, add parameter -config when running commands to configure an independent configuration file for each process.

N/A

sdkMaxLogSize

Optional

Size of an SDK log file, in bytes. The default value is 30MB.

NOTE:

This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

[10MB, 100MB]

utilLogBackups

Optional

Maximum number of obsutil log files that can be retained. The default value is 10.

N/A

utilLogLevel

Optional

obsutil log level. Possible values are:

  • DEBUG
  • INFO
  • WARN
  • ERROR

The default value is INFO.

N/A

utilLogPath

Optional

Absolute path of obsutil logs. The value must be a file path. The default value is the path of the obsutil.log file in the subfolder .obsutil_log of the user's home directory (HOME in Linux or macOS and C:\Users\<Username> in Windows).

NOTE:
  • If this parameter is left blank, no obsutil log is generated.
  • The path must be a file path and cannot be a folder path.
  • After the obsutil log function is enabled, all logs generated during commands executing are saved in the obsutil log file for problem analysis and location.
  • Ensure that the user who runs the command has the read and write permissions on the path.
NOTICE:

If multiple obsutil processes are running at the same time, log files may fail to be written concurrently. In this case, add parameter -config when running commands to configure an independent configuration file for each process.

N/A

utilMaxLogSize

Optional

Size of an obsutil log file, in bytes. The default value is 30MB.

NOTE:

This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

[10MB, 100MB]

writeBufferIoSize

Optional

Size of the cache for downloading data, in bytes. The default value is 65536.

NOTE:
  • Set this parameter based on site requirements. If the size of the file to be downloaded is large, set this parameter to a large value.
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

N/A

readBufferIoSize

Optional

Size of the cache for uploading data, in bytes. The default value is 8192.

NOTE:
  • Set this parameter based on site requirements. If a large number of small files are uploaded, set this parameter to a small value. If large files are uploaded, set this parameter to a large value.
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

[4096, 65536]

recordMaxLogSize

Optional

Size of a result list containing success, failure, or warning lists in a batch task, in bytes. The default value is 30MB.

NOTE:

This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

[5MB, 100MB]

recordBackups

Optional

Maximum number of result lists of successful or failed batch tasks that can be retained. The default value is 1024.

N/A

humanReadableFormat

Optional

Indicates whether to convert the number of bytes in the object listing result and result list content to the human-readable format. The default value is true.

N/A

showProgressBar

Optional

Indicates whether to display the progress bar on the console. The value true indicates that the progress bar is displayed. The default value is true.

N/A

showStartTime

Optional

Indicates whether to display the start time on the console. The value true indicates that start time is displayed. The default value is true.

N/A

helpLanguage

Optional

Language of the help documents. Options are as follows:

  • Chinese
  • English

The default value is English.

N/A

defaultTempFileDir

Optional

Indicates the directory for storing temporary files during download.

The default value is the .obsutil_tempfile subfolder in the user directory (HOME in Linux or macOS and C:\Users\<Username> in Windows).

NOTE:
  • Temporary files generated during multipart download are stored in this directory. Therefore, ensure that the user who executes obsutil has the write permission on the path.
  • The available space of the partition where the path is located must be greater than the size of the objects to be downloaded.

N/A

checkSourceChange

Optional

Indicates whether to check the change of source files or objects during upload/download/copy. The value true indicates that the function is enabled.

N/A

skipCheckEmptyFolder

Optional

Indicates whether to skip checking empty folders on the OBS server during download. true indicates to skip the check. The default value is false.

NOTICE:

If this parameter is set to true, the directory structure downloaded to your local PC may be different from that in OBS.

N/A

fsyncForDownload

Optional

Indicates whether to forcibly synchronize memory data to disks during download. The value true indicates to enable forcible synchronization. The default value is false.

NOTE:
  • Set this parameter to true for scenarios that require high data reliability.
  • If this parameter is set to true, the download performance will be deteriorated. Therefore, exercise caution when using this parameter.

N/A

memoryEconomicalScanForUpload

Optional

Indicates whether to use the scanning mode that occupies less memory space when uploading a folder. The value true indicates using this method. The default value is true.

N/A

forceOverwriteForDownload

Optional

Indicates to forcibly overwrite the local executable file (even if the local executable file is running) when downloading objects to the Linux OS or macOS. The value true means to overwrite, and the default value is true.

N/A

panicForSymbolicLinkCircle

Optional

Indicates the processing method after a symbolic link loop is detected during upload. The value false indicates that errors are only recorded. The value true indicates that panic is triggered. The default value is false.

N/A

fastFailThreshold

Optional

Threshold for fast failure upon 4XX errors of batch tasks. When the number of 4XX errors exceeds the threshold, the fast failure process is triggered. All tasks that are not executed or being scanned are suspended. The default value is 5.

NOTE:

The fast failure mechanism is to avoid excessive traffic generated during batch task execution. To start a fast failure as soon as possible, set this parameter to 0 or -1, indicating that the fast failure process starts immediately whenever a 4XX error occurs.

N/A

abortHttpStatusForResumableTasks

Optional

HTTP status codes for fast interruption of multipart upload, download, and copy tasks. If a sub-task of a multipart task receives an HTTP code that falls into this range, the multipart task is immediately interrupted. The default value is 401,403,404,405,409.

NOTE:
  • Multiple HTTP status codes can be carried and separated by commas (,), for example, 401,403,404.
  • The status code must be a 4XX HTTP status code. Other status codes are ignored.

Default value

showBytesForCopy

Optional

Indicates whether the progress bar displays the rate in bytes when objects are copied between buckets. The default value is false.

N/A

proxyUrl

Optional

HTTP proxy example: http://username:password@your-proxy:8080

NOTE:
  • HTTP proxy format: http://[Username:Password@]Proxy server address:Port number. The Username and Password are optional.
  • The proxyUrl parameter and system environment variables are in the following priority order: proxyUrl > HTTPS_PROXY > HTTP_PROXY.
  • The user name and password cannot contain colons (:) and at signs (@), which will result in parsing errors.

N/A

faultTolerantMode

Optional

Indicates whether to ignore the panic caused by read operations. Default value false indicates that the panic is not ignored.

N/A
  • Set parameters with N/A as the recommended value based on your needs.
  • You are advised to specify sdkLogPath and utilLogPath to enable SDK logging and obsutil logging.
  • The values of defaultBigfileThreshold, defaultPartSize, rateLimitThreshold, sdkMaxLogSize, utilMaxLogSize, recordMaxLogSize, readBufferIoSize, and writeBufferIoSizecan can contain a unit, for example, 1MB (indicating 1048576 bytes).