Using obsutil for Resumable Data Transfer

Updated on 2025-02-10 GMT+08:00

View PDF

obsutil supports resumable data transfer (upload, download, and copy) for large files by using the multipart algorithms for upload, download, and copy. You can set the threshold size for starting a multipart upload, download, or copy task based on your actual requirements to resume the upload, download, or copy task if the task fails or is interrupted. You can specify the threshold size for starting a multipart task in either of the following ways:

Method 1

Set defaultBigfileThreshold in the configuration file. For details, see Configuration Parameters.

Method 2

Set threshold, a command-level parameter, when you run commands for object uploads, object downloads, object copy, synchronous uploads of incremental objects, synchronous downloads of incremental objects, and synchronous copy of incremental objects.

Example: obsutil cp d:\temp\test.txt obs://bucket-test/key -threshold=52428800

In this command:

obsutil cp d:\temp\test.txt obs://bucket-test/key uploads file test.txt in the temp directory under D drive to bucket bucket-test and renames the file key.
-threshold=52428800 starts a multipart upload when the threshold 50 MB is reached.

The following example is based on a Windows OS:

obsutil cp d:\temp\test.txt obs://bucket-test/key -threshold=52428800

Parallel:      3                   Jobs:          3
Threshold:     50.00MB             PartSize:      auto
VerifyLength:  false               VerifyMd5:     false
CheckpointDir: xxxx

[====================================================] 100.00% 1.68 MB/s 5s
Upload successfully, 8.46MB, d:\temp\test.txt --> obs://bucket-test/key

NOTE:

Priority: Command level parameter threshold has higher priority than the defaultBigfileThreshold in the configuration file.
The threshold size of a multipart task applies to single files or objects. When the size of a file or object is greater than the threshold value, the multipart algorithm is applied to the file or object.
The multipart algorithm and resumable data transfer are forcibly bound together. That is, once the multipart algorithm is used, the resumable data transfer is enabled for the task.

Parent topic: Best Practices

Previous topic: Fine-Tuning obsutil Performance

Next topic: Using obsutil to Upload a Symbolic Link

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot