Copied.
Resumable Upload
Scenarios
Uploading large files often fails due to poor network conditions or program breakdowns. Re-uploads not only waste resources, but may also fail again. To avoid such issues, you can use the resumable upload API, which works by dividing a file into multiple parts and then uploading them separately. This API helps save resources for re-uploads and improves efficiency by concurrent uploads.
Important Notes
- To upload objects, you must be the bucket owner or have the required permission (obs:object:PutObject in IAM or bPutObjectb in a bucket policy). For details, see Introduction to OBS Access Control, IAM Custom Policies, and Configuring an Object Policy.
- If you want to know the regions where OBS is available and the mapping between regions and endpoints, refer to Regions and Endpoints.
- The file uploaded by the resumable upload API must be larger than 100 KB in size.
- Before using SDK for resumable upload, you must enable the resumable unload option. Only in this way, the progress of the last upload can be read when you continue the upload process again.
Ways to Upload
You can use SDKs, OBS Browser+, or obsutil to upload objects.
Using the GUI Tool - OBS Browser+
- Log in to OBS Browser+.
- Click the bucket where you want to upload files or folders.
- Click Upload and then Add File or Folder, as shown in Figure 1.
For better experience when using the Add File function, you are advised to upload a maximum of 100 files at a time. If you need to upload more, place all the files in a folder and upload them by adding a folder.
- If message "Service Unavailable" is displayed when files are being uploaded, try again later.
- If an access deny message is displayed when you are uploading a file or folder, possible causes are as follows:
- Access to the bucket is restricted by an ACL. For example, you do not have the write permission for the bucket.
- Access to the bucket is restricted by a bucket policy. For example, you do not have the write permission for the bucket, or write operations cannot be performed on the bucket during the current period.
If such message is displayed, check ACL and policy settings of the bucket and resolve the problem accordingly.
You must have access to the file you want to upload, or the file upload will fail.
- In the displayed dialog box, select the file or folder you want to upload and click Open.
You can upload one folder or multiple files at a time. To upload multiple files, hold down Ctrl or Shift to select multiple files and batch upload them. You can also press Ctrl+A to select all files. The operations are consistent with those in Windows operating systems.
- Click OK.
Using the CLI Tool - obsutil
Command Line Structure
- In Windows
- Uploading a file
obsutil cp file_url obs://bucket[/key] [-arcDir=xxx] [-dryRun] [-link] [-u] [-vlength] [-vmd5] [-p=1] [-threshold=5248800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-o=xxx] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
- Uploading a folder
obsutil cp folder_url obs://bucket[/key] -r [-arcDir=xxx] [-dryRun] [-link] [-f] [-flat] [-u] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
- Uploading multiple files/folders
obsutil cp file1_url,folder1_url|filelist_url obs://bucket[/prefix] -msm=1 [-r] [-arcDir=xxx] [-dryRun] [-link] [-f] [-u] [-vlength] [-vmd5] [-flat] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx][-timeRange=time1-time2] [-at] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
In this command, /prefix is the name prefix for uploading folders. For the execution examples, see Upload.
- Uploading a file
- In Linux or macOS
- Uploading a file
./obsutil cp file_url obs://bucket[/key] [-arcDir=xxx] [-dryRun] [-link] [-u] [-vlength] [-vmd5] [-p=1] [-threshold=5248800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-o=xxx] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
- Uploading a folder
./obsutil cp folder_url obs://bucket[/key] -r [-arcDir=xxx] [-dryRun] [-link] [-f] [-flat] [-u] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
- Uploading multiple files/folders
./obsutil cp file1_url,folder1_url|filelist_url obs://bucket[/prefix] -msm=1 [-r] [-arcDir=xxx] [-dryRun] [-link] [-f] [-u] [-vlength] [-vmd5] [-flat] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx][-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
In this command, /prefix is the name prefix for uploading folders. For the execution examples, see Upload.
- Uploading a file
Examples
- Take the Windows OS as an example. Run the obsutil cp d:\temp\test.txt obs://bucket-test/key command to upload the test.txt file in the temp directory in the D: drive to bucket bucket-test and rename the file as key.
obsutil cp d:\temp\test.txt obs://bucket-test/key Parallel: 5 Jobs: 5 Threshold: 50.00MB PartSize: auto VerifyLength: false VerifyMd5: false CheckpointDir: C:\Users\Administrator\.obsutil_checkpoint [====================================================] 100.00% 1.68 MB/s 8.46MB/8.46MB 5s Upload successfully, 8.46MB, n/a, d:\temp\test.txt --> obs://bucket-test/key, cost [5], status [200], request id [0000016979E1D2B2860BB5181229C72C]
- Take the Windows OS as an example. Run the obsutil cp d:\temp obs://bucket-test -f -r command to recursively upload all files and subfolders in the temp directory in the D: drive to the temp folder in bucket bucket-test.
obsutil cp d:\temp obs://bucket-test -f -r Parallel: 5 Jobs: 5 Threshold: 50.00MB PartSize: auto VerifyLength: false VerifyMd5: false CheckpointDir: C:\Users\Administrator\.obsutil_checkpoint Task id: 104786c8-27c2-48fc-bc6a-5886596fb0ed OutputDir: C:\Users\Administrator\.obsutil_output [========================================================] 100.00% tps:35.71 2.02 KB/s 7.20MB/7.20MB 0s Succeed count is: 5 Failed count is: 0 Metrics [max cost:90 ms, min cost:45 ms, average cost:63.80 ms, average tps:35.71] Task id: 104786c8-27c2-48fc-bc6a-5886596fb0ed
- For more examples, see Upload.
Parameter Description
Parameter |
Optional or Mandatory |
Description |
---|---|---|
file_url |
Optional for uploading multiple files/folders Mandatory for uploading a file |
Local file path
NOTE:
|
folder_url |
Optional for uploading multiple files/folders Mandatory for uploading a folder |
Local folder path
NOTE:
|
filelist_url |
Optional for uploading multiple files/folders |
Indicates the path of the file that contains the list of files/folders to be uploaded. If this parameter is configured, msm must be set to 2. |
bucket |
Mandatory |
Bucket name |
key |
Optional |
Indicates the object name or object name prefix specified when uploading a file, or the object name prefix specified when uploading a folder. The rules are as follows:
NOTE:
For details about how to use this parameter, see Upload. |
fr |
Optional for uploading a file (additional parameter) |
Generates an operation result list when uploading a file. |
flat |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Uploads all files in a folder but not the folder itself. |
arcDir |
Optional (additional parameter) |
Path to which the uploaded files are archived |
dryRun |
Optional (additional parameter) |
Conducts a dry run. |
link |
Optional (additional parameter) |
Uploads the actual path of the symbolic-link file/folder
NOTICE:
|
u |
Optional (additional parameter) |
Indicates incremental upload. If this parameter is set, each file can be uploaded only when it does not exist in the bucket, its size is different from the namesake one in the bucket, or it has the latest modification time.
CAUTION:
When you compare each local file with data in the bucket, a billable HEAD request is generated. For details, see Requests. |
vlength |
Optional (additional parameter) |
After the upload completes, check whether the sizes of the objects in the bucket are the same as those of the local files. |
vmd5 |
Optional (additional parameter) |
After the upload completes, check whether the MD5 values of the objects in the bucket are the same as those of the local files.
NOTE:
|
p |
Optional (additional parameter) |
Indicates the maximum number of concurrent multipart upload tasks when uploading a file. The default value is the value of defaultParallels in the configuration file. |
threshold |
Optional (additional parameter) |
Indicates the threshold for enabling multipart upload, in bytes. The default value is the value of defaultBigfileThreshold in the configuration file.
NOTE:
|
acl |
Optional (additional parameter) |
Access control policies that can be specified when uploading files. Possible values are:
NOTE:
The preceding four values indicate private read and write, public read, public read and write, and bucket owner full control. |
sc |
Optional (additional parameter) |
Indicates the storage classes of objects that can be specified when uploading files. Possible values are:
|
meta |
Optional (additional parameter) |
Indicates the standard and custom metadata that can be specified during file upload. This parameter should be configured in the following format: key1:value1#key2:value2#key3:value3.
NOTE:
|
ps |
Optional (additional parameter) |
Indicates the size of each part in a multipart upload task, in bytes. The value ranges from 100KB to 5GB. The default value is the value of defaultPartSize in the configuration file.
NOTE:
|
cpd |
Optional (additional parameter) |
Indicates the folder where the part records reside. The default value is .obsutil_checkpoint, the subfolder in the home directory of the user who executes obsutil commands.
NOTE:
A part record is generated during a multipart upload and saved to the upload subfolder. After the upload succeeds, its part record is deleted automatically. If the upload fails or is suspended, the system attempts to resume the task according to its part record when you perform the upload the next time. |
r |
Mandatory for uploading a folder (additional parameter) Optional for uploading multiple files/folders |
Indicates files and subfolders within the folder when uploading a folder recursively. |
f |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Runs in force mode. |
j |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Indicates the maximum number of concurrent tasks for uploading a folder. The default value is the value of defaultJobs in the configuration file.
NOTE:
The value is ensured to be greater than or equal to 1. |
msm |
Mandatory for uploading multiple files/folders (additional parameter) |
Enables the mode for uploading multiple files/folders. Possible values are 1 and 2.
NOTE:
|
exclude |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Indicates the file matching patterns that are excluded, for example: *.txt.
NOTE:
NOTICE:
|
include |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Indicates the file matching patterns that are included, for example: *.jpg.
NOTE:
NOTE:
Example of uploading files in a request with the include parameter contained: ./obsutil cp /localpath/ obs://test/ -include=/localpath/2022-12-09/* -f -r This command uploads files that are under localpath and start with /localpath/2022-12-09/ to bucket test.
NOTICE:
|
at |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Indicates that only the files whose latest access time is within the value of timeRange are uploaded.
NOTE:
|
disableDirObject |
Optional for uploading multiple folders (additional parameter) |
Indicates the folders themselves are not uploaded as an object. Configuring this parameter can avoid uploading empty folders to a bucket. If a folder contains files, the files will be uploaded and the original path format is retained. |
timeRange |
Optional for uploading a folder or multiple files/folders (additional parameter) |
Indicates the time range matching pattern when uploading files. Only files whose latest modification time is within the configured time range are uploaded. This pattern has a lower priority than the file matching patterns (exclude/include). That is, the time range matching pattern is executed after the configured file matching patterns.
NOTE:
NOTICE:
Time in the matching pattern is the UTC time. |
mf |
Optional (additional parameter) |
Indicates that the name matching pattern (include or exclude) and the time matching pattern (timeRange) also take effect on folders. |
o |
Optional (additional parameter) |
Indicates the folder where operation result lists reside. After the command is executed, result lists (possibly including success, failure, and warning files) are generated in the folder. The default value is .obsutil_output, the subfolder in the home directory of the user who executes obsutil commands.
NOTE:
|
config |
Optional (additional parameter) |
User-defined configuration file for executing the current command. To learn the parameters that can be configured in this file, see Configuration Parameters. |
e |
Optional (additional parameter) |
Specifies the endpoint. |
i |
Optional (additional parameter) |
Specifies the user's AK. |
k |
Optional (additional parameter) |
Specifies the user's SK. |
t |
Optional (additional parameter) |
Specifies the user's security token. |
Response
Field |
Description |
---|---|
Parallel |
Parameter -p in the request |
Jobs |
Parameter -j in the request |
Threshold |
Parameter -threshold in the request |
PartSize |
Parameter -ps in the request |
Exclude |
Parameter -exclude in the request |
Include |
Parameter -include in the request |
TimeRange |
Parameter -timeRange in the request |
VerifyLength |
Parameter -vlength in the request |
VerifyMd5 |
Parameter -vmd5 in the request |
CheckpointDir |
Parameter -cpd in the request |
OutputDir |
Parameter -o in the request |
ArcDir |
Parameter -arcDir in the request |
Succeed count |
Number of successful tasks |
Failed count |
Number of failed tasks |
Skip count |
Number of tasks that are skipped during incremental upload, download, or copy, and synchronous upload, download, or copy.
NOTE:
Skipped tasks are recorded into successful tasks. |
Warning count |
Number of tasks that are executed successfully but contain warnings.
NOTE:
|
Succeed bytes |
Number of bytes that are successfully uploaded or downloaded. |
max cost |
Maximum duration of all tasks, in ms |
min cost |
Minimum duration of all tasks, in ms |
average cost |
Average duration of all tasks, in ms |
average tps |
The average number of tasks completed per second |
Task id |
Unique ID of an operation, which is used to search for the result list generated in a batch task |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot