Help Center/ Object Storage Service/ User Guide/ Object Management/ Upload/ Multipart Upload
Updated on 2024-10-24 GMT+08:00
View PDF
Share

Multipart Upload

Scenarios

Multipart upload allows you to upload a single object as a group of parts. Each part is a contiguous one of the object data. You can upload these parts separately and in any sequence. If a part fails to be uploaded, you can upload it again without affecting other parts. After all parts are uploaded, OBS assembles these parts to create the object.

Generally, if the size of an object reaches 100 MB, multipart upload is recommended. For example, if you want to upload a 500 MB object to an OBS bucket, you can use OBS Browser+ for multipart upload. OBS Browser+ divides the object into small parts and then uploads the parts. Alternatively, you can call the multipart upload API, improving upload efficiency and reducing failures.

Advantages of Multipart Upload

  • Improved throughput: You can upload parts in parallel to improve throughput.
  • Quick recovery from network errors: Small parts minimize the impact of restarting a failed uploading caused by network errors.
  • Convenient pause and resuming of object uploads: You can upload parts at any time. A multipart upload will not expire after being initiated. You must explicitly complete or abort a multipart upload.
  • Starting uploading before knowing the object size: You can upload an object while creating it.

Constraints

Table 1 Constraints on multipart upload

Item

Constraints

Maximum object size

48.8 TB

Maximum number of parts for each upload task

10,000

Part number

1 to 10,000 (inclusive)

Part size

5 MB to 5 GB. The size of the last part is between 0 bytes to 5 GB.

Maximum number of parts returned in response to the request for listing uploaded parts

1,000

Maximum number of multipart uploads returned in response to the request for listing initiated multipart uploads

1,000

If you have over 48.8 TB data to upload, refer to Migrating Local Data to OBS.

Multipart Upload Process

Figure 1 Multipart upload
  1. Divide a file to be uploaded into parts.
  2. Initiate a multipart upload.

    When you send a request to start multipart upload, OBS returns a response with the upload ID, which is the unique identifier of the multipart upload. This ID must be included in the request for uploading parts, listing uploaded parts, completing a multipart upload, or aborting a multipart upload.

  3. Upload parts.

    When uploading a part, you must specify the upload ID and a part number. You can select any part number between 1 and 10,000. A part number uniquely identifies a part and its location in the object you are uploading. If the number of an uploaded part is used to upload a new part, the uploaded part will be overwritten.

    Whenever you upload a part, OBS returns the ETag header in the response. For each part upload, you must record the part number and the ETag value. These part numbers and ETag values are required in subsequent requests to complete the multipart upload task.

    When concurrent upload operations are performed for the same part of an object, the server complies with the Last Write Wins policy, but the time referred in Last Write is the time when the part metadata is created. To ensure data accuracy, the client must be locked during the concurrent upload for the same part of an object. Concurrent upload for different parts of an object does not require the client to be locked.

  4. (Optional) Copy parts.

    After initiating a multipart upload, you can specify upload ID for the multipart upload and call the multipart copy API to copy part or all of the uploaded object as parts.

    If you copy the source object as a part called part1 and another part1 already exists before the copy operation, the original part1 will be overwritten by the new one after the copy operation. After the copy succeeds, only the new part1 can be listed and the original part1 will be deleted. Therefore, ensure that the original part does not exist or has no value when copying a part. Otherwise, data may be deleted by mistake. The source object does not change during the copy.

    You cannot determine whether a request is successful only based on the status_code in the returned HTTP header. If 200 is returned for status_code, the server has received the request and started to process the request. The copy is successful only when the body in the response contains ETag.

  5. (Optional) Abort the multipart upload.

    You can abort a multipart upload. After a multipart upload is aborted, the upload ID cannot be used to upload any part. Then, OBS releases the storage of all uploaded parts. If you stop an ongoing multipart upload, the uploading will still complete and the result can be successful or failed. To release the storage capacity occupied by all parts, you need to abort the multipart upload after the entire task is complete.

  6. (Optional) List parts.
    • Listing uploaded parts

      You can list the parts of a specific multipart upload task or the parts of all the multipart upload tasks in progress. Information about uploaded parts in a specific multipart upload will be returned for a request to list uploaded parts. For each request to list uploaded parts, OBS returns information about the uploaded parts in the specific multipart upload. Information about a maximum of 1,000 parts can be returned. If there are more than 1,000 parts in a multipart upload, you need to send multiple requests to list all uploaded parts. The list of uploaded parts does not include assembled parts.

      A returned list can only be used for verification. After a multipart upload is complete, the result in the list is no longer valid. However, when part numbers and the ETag values returned by OBS are uploaded, the list of your specified part numbers will be reserved.

    • Listing multipart uploads

      You can list initiated multipart uploads by listing the multipart uploads in the bucket. Initiated multipart uploads refer to the multipart uploads that are not assembled or aborted after initiation. A maximum of 1,000 multipart uploads can be returned for each request. If there are more than 1,000 multipart uploads in progress, you need to send more requests to query the remaining tasks.

  7. Assemble parts.

    OBS creates an object by assembling the parts in ascending order based on the part number. If any object metadata is provided for initiating a multipart upload task, OBS associates the metadata with the object. After the request for the multipart upload is complete, the parts will no longer exist. A part assembling request must contain the upload ID, part numbers, and a list of corresponding ETag values. An OBS response includes an ETag that uniquely identifies the combined object data. The ETag is not an MD5 hash of the object data.

    • After the multipart upload task is initiated and one or more parts are uploaded, you must assemble the parts or abort the multipart upload. Otherwise, you have to pay for the storage fee of the uploaded parts. OBS releases the storage and stops charging the storage fee only after the uploaded parts are assembled or the multipart upload is aborted.
    • If 10 parts are uploaded but only nine parts are assembled, the parts that are not assembled will be automatically deleted by the system and cannot be restored after being deleted. Before assembling the parts, adopt the API used to list the parts that have been uploaded to check all parts to ensure that no part is missed.

Permissions

You can perform multipart upload only after being granted with the permission. You can use ACLs, bucket policies, or user policies to grant users the permission. The following table lists multipart upload operations and the required permissions that can be granted by ACLs, bucket policies, or user policies.

Operation

Required Permissions

Initiate a multipart upload.

To perform this operation, you need to have the PutObject permission.

A bucket owner can grant the PutObject permission to others.

Upload parts.

To perform this operation, you need to have the PutObject permission.

Only the initiator of a multipart upload can upload parts. The bucket owner must grant the multipart upload initiator the PutObject permission so that the initiator can upload parts of the object.

Copy parts.

To perform this operation, you need to have the PutObject permission as well as the GetObject permission on the object to be copied.

Only the initiator of a multipart upload can copy parts. The bucket owner must grant the multipart upload initiator the PutObject permission so that the initiator can upload parts of the object.

Assemble parts.

To perform this operation, you need to have the PutObject permission.

Only the initiator of a multipart upload can assemble parts. The bucket owner must grant the multipart upload initiator the PutObject permission so that the initiator can complete the multipart upload.

Abort the multipart upload.

To perform this operation, you need to have the AbortMultipartUpload permission.

By default, only the bucket owner and the multipart upload initiator have this permission. In addition to the default configuration, the bucket owner can allow trustees to perform this operation. The bucket owner can also deny any trustees performing this operation.

List uploaded parts.

To perform this operation, you need to have the ListMultipartUploadParts permission.

By default, the bucket owner can list the uploaded parts of any multipart upload to the bucket. The multipart upload initiator can list the uploaded parts of a specific multipart upload.

In addition to the default configuration, the bucket owner can allow trustees to perform this operation. The bucket owner can also deny any trustees performing this operation.

List multipart uploads.

To list multipart uploads to the bucket, you need to have the ListBucketMultipartUploads permission.

In addition to the default configuration, the bucket owner can allow trustees to perform this operation.

Important Notes

  • A directory cannot be uploaded. Only one object can be uploaded at a time.
  • Each request can upload only one part, but multiple requests can be initiated at the same time.
  • If you want to upload a large number of Deep Archive objects, you can upload them in the Standard storage class and then transition them to the Deep Archive storage class through lifecycle rules to lower the costs on PUT requests.
  • If a large number of objects need to be uploaded, do not name the objects using sequential prefixes, such as timestamps or alphabetical order. Objects named with sequential prefixes may be stored in a specific storage partition. In such case, if there are a large number of access requests for the objects, the requests cannot be handled efficiently.
  • It is recommended to enable versioning to prevent objects with the same name from being overwritten. Versioning keeps every version of objects in the same bucket. You can restore any historical version at any time.
  • The checkpoint file records the status of uploads using SDKs. Ensure that you have write permissions for the checkpoint file.
  • Do not modify the verification information in the checkpoint file. If the checkpoint file is damaged, all fragments will be uploaded again.
  • If the local file changes during the upload, all fragments will be uploaded again.

Ways to Upload

You can use OBS Console, APIs, SDKs, OBS Browser+, or obsutil to upload objects.

Using OBS Console

  1. In the navigation pane of OBS Console, choose Object Storage.
  2. In the bucket list, click the bucket you want to operate.
  3. Go to the folder that you want to upload files to and click Upload Object. The Upload Object dialog box is displayed.

    Batch upload is used as an example here. If a region only supports single upload, follow the on-screen instructions.

    If the files that you want to upload to OBS are stored in Microsoft OneDrive, it is recommended that the names of these files contain a maximum of 32 characters to ensure compatibility.

    Figure 2 Object Upload

  4. Select a storage class. If you do not specify a storage class, the objects you upload inherit the default storage class of the bucket.

    An object can have a different storage class from its bucket. You can specify a storage class for an object when uploading it, or you can change the object storage class after the object is uploaded.

  5. Drag and drop the files or folders you want to upload to the Upload Object area.

    You can also click add files in the Upload Object area to select files.

  6. Server-Side Encryption: Choose Disable, SSE-KMS, or SSE-OBS. For details, see Server-Side Encryption.

    If the bucket has Server-Side Encryption configured, an object you upload will inherit encryption from the bucket by default, but you can change the encryption option as required.

  7. (Optional) To configure WORM retention policies, or metadata, click Next: (Optional) Configure Advanced Settings.

    WORM retention policies can be configured in the advanced settings only when WORM is enabled for the bucket.

    Configuring metadata: Add metadata ContentDisposition, ContentLanguage, WebsiteRedirectLocation, ContentEncoding, or ContentType as needed. For more information, see OBS Metadata. Metadata is a set of name-value pairs. The metadata value cannot be left blank. You can add two or more metadata entries by clicking Add.

    Configuring WORM retention: Choose Inherit from bucket, or choose Configure and then specify a retention period, to automatically protect new objects uploaded to the bucket from being deleted.

    Figure 3 Configuring metadata or WORM retention

  8. Click Upload.

Using APIs

Operations on Multipart Upload

Using SDKs

Java

Python: not supported

C

Go: not supported

BrowserJS

.NET

Android

iOS

PHP

Node.js

Using the GUI Tool - OBS Browser+

  1. Log in to OBS Browser+.
  2. Click the bucket where you want to upload files or folders.
  3. Click Upload and then Add File or Folder, as shown in Figure 4.

    Figure 4 Uploading a file or folder

    For better experience when using the Add File function, you are advised to upload a maximum of 100 files at a time. If you need to upload more, place all the files in a folder and upload them by adding a folder.

    1. If message "Service Unavailable" is displayed when files are being uploaded, try again later.
    2. If an access deny message is displayed when you are uploading a file or folder, possible causes are as follows:
    • Access to the bucket is restricted by an ACL. For example, you do not have the write permission for the bucket.
    • Access to the bucket is restricted by a bucket policy. For example, you do not have the write permission for the bucket, or write operations cannot be performed on the bucket during the current period.

      If such message is displayed, check ACL and policy settings of the bucket and resolve the problem accordingly.

    You must have access to the file you want to upload, or the file upload will fail.

  4. In the displayed dialog box, select the file or folder you want to upload and click Open.

    You can upload one folder or multiple files at a time. To upload multiple files, hold down Ctrl or Shift to select multiple files and batch upload them. You can also press Ctrl+A to select all files. The operations are consistent with those in Windows operating systems.

  5. Click OK.

Using the CLI Tool - obsutil

Command Line Structure

  • In Windows
    • Uploading a file 
      obsutil cp file_url obs://bucket[/key] [-arcDir=xxx] [-dryRun] [-link] [-u] [-vlength] [-vmd5] [-p=1] [-threshold=5248800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-o=xxx] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Uploading a folder 
      obsutil cp folder_url obs://bucket[/key] -r [-arcDir=xxx] [-dryRun] [-link] [-f] [-flat] [-u] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Uploading multiple files/folders 
      obsutil cp file1_url,folder1_url|filelist_url obs://bucket[/prefix] -msm=1 [-r] [-arcDir=xxx] [-dryRun] [-link] [-f] [-u] [-vlength] [-vmd5] [-flat] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx][-timeRange=time1-time2] [-at] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]

      In this command, /prefix is the name prefix for uploading folders. For the execution examples, see Upload.

  • In Linux or macOS
    • Uploading a file 
      ./obsutil cp file_url obs://bucket[/key] [-arcDir=xxx] [-dryRun] [-link] [-u] [-vlength] [-vmd5] [-p=1] [-threshold=5248800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-o=xxx] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Uploading a folder 
      ./obsutil cp folder_url obs://bucket[/key] -r [-arcDir=xxx] [-dryRun] [-link] [-f] [-flat] [-u] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Uploading multiple files/folders 
      ./obsutil cp file1_url,folder1_url|filelist_url obs://bucket[/prefix] -msm=1 [-r] [-arcDir=xxx] [-dryRun] [-link] [-f] [-u] [-vlength] [-vmd5] [-flat] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx][-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]

      In this command, /prefix is the name prefix for uploading folders. For the execution examples, see Upload.

Examples

  • Take the Windows OS as an example. Run the obsutil cp d:\temp\test.txt obs://bucket-test/key command to upload the test.txt file in the temp directory in the D: drive to bucket bucket-test and rename the file as key. 
    obsutil cp d:\temp\test.txt obs://bucket-test/key

Parallel:      5                   Jobs:          5
Threshold:     50.00MB             PartSize:      auto
VerifyLength:  false               VerifyMd5:     false
CheckpointDir: C:\Users\Administrator\.obsutil_checkpoint

[====================================================] 100.00% 1.68 MB/s 8.46MB/8.46MB 5s
Upload successfully, 8.46MB, n/a, d:\temp\test.txt --> obs://bucket-test/key, cost [5], status [200], request id [0000016979E1D2B2860BB5181229C72C]
  • Take the Windows OS as an example. Run the obsutil cp d:\temp obs://bucket-test -f -r command to recursively upload all files and subfolders in the temp directory in the D: drive to the temp folder in bucket bucket-test. 
    obsutil cp d:\temp obs://bucket-test -f -r

Parallel:      5                   Jobs:          5
Threshold:     50.00MB             PartSize:      auto
VerifyLength:  false               VerifyMd5:     false
CheckpointDir: C:\Users\Administrator\.obsutil_checkpoint

Task id: 104786c8-27c2-48fc-bc6a-5886596fb0ed
OutputDir: C:\Users\Administrator\.obsutil_output

[========================================================] 100.00% tps:35.71 2.02 KB/s 7.20MB/7.20MB 0s
Succeed count is:   5         Failed count is:    0
Metrics [max cost:90 ms, min cost:45 ms, average cost:63.80 ms, average tps:35.71]

Task id: 104786c8-27c2-48fc-bc6a-5886596fb0ed
  • For more examples, see Upload.

Parameter Description

Parameter

Optional or Mandatory

Description

file_url

Optional for uploading multiple files/folders

Mandatory for uploading a file

Local file path

NOTE:
  • Do not nest paths when uploading multiple files/folders. For example, you cannot specify /a/b/c and /a/b/ at the same time.
  • If this parameter is configured when uploading multiple files/folders, msm must be set to 1. In this case, use commas (,) to separate multiple file paths, for example, file_url1,file_url2.
  • Files and folders can both be included when uploading multiple files/folders. For example, file_url1,folder_url1,file_url2,folder_url2.

folder_url

Optional for uploading multiple files/folders

Mandatory for uploading a folder

Local folder path

NOTE:
  • If flat is not configured when uploading a folder, the entire folder is uploaded. If flat is configured, all files in the folder are uploaded.
  • Do not nest paths when uploading multiple files/folders. For example, you cannot specify /a/b/c and /a/b/ at the same time.
  • If this parameter is configured when uploading multiple files/folders, msm must be set to 1. In this case, use commas (,) to separate multiple folder paths, for example, folder_url1,folder_url2.
  • Files and folders can both be included when uploading multiple files/folders. For example, file_url1,folder_url1,file_url2,folder_url2.

filelist_url

Optional for uploading multiple files/folders

Indicates the path of the file that contains the list of files/folders to be uploaded. If this parameter is configured, msm must be set to 2.

NOTE:
  • The list file is in common text file formats, such as TXT and CSV. Each line in the file indicates a file or folder to be uploaded. For example:

    file_url1

    file_url2

    folder_url1

    folder_url2

  • Do not nest paths in the list file. For example, you cannot specify /a/b/c and /a/b/ at the same time.

bucket

Mandatory

Bucket name

key

Optional

Indicates the object name or object name prefix specified when uploading a file, or the object name prefix specified when uploading a folder.

The rules are as follows:

  • If this parameter is left blank when uploading a file, the file is uploaded to the root directory of the bucket and the object name is the file name. If the value ends with a slash (/), the value is used as the object name prefix when the file is uploaded, and the object name is the value plus the file name. If the value does not end with a slash (/), the file is uploaded with the value as the object name.
  • If this parameter is left blank when uploading a folder, the folder is uploaded to the root directory of the bucket. If the value ends with a slash (/), the value is used as the object name prefix of the folder to be uploaded. If the value does not end with a slash (/), the folder to be uploaded is prefixed with the value plus a slash (/).
NOTE:

For details about how to use this parameter, see Upload.

fr

Optional for uploading a file (additional parameter)

Generates an operation result list when uploading a file.

flat

Optional for uploading a folder or multiple files/folders (additional parameter)

Uploads all files in a folder but not the folder itself.

arcDir

Optional (additional parameter)

Path to which the uploaded files are archived

dryRun

Optional (additional parameter)

Conducts a dry run.

link

Optional (additional parameter)

Uploads the actual path of the symbolic-link file/folder

NOTICE:
  • If this parameter is not specified and the file to be uploaded is a symbolic-link file whose target file does not exist, the exception message "The system cannot find the file specified" will be displayed in Windows OS, while the exception message "No such file or directory" will be displayed in macOS or Linux OS.
  • Avoid the symbolic link loop of a folder, otherwise, the upload will exit due to panic. If you do not want the system to panic, set panicForSymbolicLinkCircle to false in the configuration file.

u

Optional (additional parameter)

Indicates incremental upload. If this parameter is set, each file can be uploaded only when it does not exist in the bucket, its size is different from the namesake one in the bucket, or it has the latest modification time.

CAUTION:

When you compare each local file with data in the bucket, a billable HEAD request is generated. For details, see Requests.

vlength

Optional (additional parameter)

After the upload completes, check whether the sizes of the objects in the bucket are the same as those of the local files.

vmd5

Optional (additional parameter)

After the upload completes, check whether the MD5 values of the objects in the bucket are the same as those of the local files.

NOTE:
  • If the size of the file or folder to be uploaded is too large, using this parameter will degrade the overall performance due to MD5 calculation.
  • After the MD5 verification is successful, this parameter value is used for metadata x-obs-meta-md5chksum, for later MD5 verification during download or copy.

p

Optional (additional parameter)

Indicates the maximum number of concurrent multipart upload tasks when uploading a file. The default value is the value of defaultParallels in the configuration file.

threshold

Optional (additional parameter)

Indicates the threshold for enabling multipart upload, in bytes. The default value is the value of defaultBigfileThreshold in the configuration file.

NOTE:
  • If the size of the file or folder to be uploaded is smaller than the threshold, upload it directly. Otherwise, a multipart upload is required.
  • If you upload a file or folder directly, no part record is generated, and resumable transmission is not supported.
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

acl

Optional (additional parameter)

Access control policies that can be specified when uploading files. Possible values are:

  • private
  • public-read
  • public-read-write
  • bucket-owner-full-control
NOTE:

The preceding four values indicate private read and write, public read, public read and write, and bucket owner full control.

sc

Optional (additional parameter)

Indicates the storage classes of objects that can be specified when uploading files. Possible values are:

  • standard: Standard storage class. It features low access latency and high throughput, and is applicable to storing frequently accessed data (multiple accesses per month) or data that is smaller than 1 MB.
  • warm: Infrequent Access storage class. It is ideal for storing infrequently accessed (less than 12 times a year) data, but when needed, the access has to be fast.
  • cold: Archive storage class. It provides secure, durable, and inexpensive storage for rarely-accessed (once a year) data.
  • deep-archive: Deep Archive storage class (under limited beta testing). It is suitable for storing data that is barely (once every few years) accessed. This storage class costs less than the Archive storage class, but takes longer time (usually several hours) to restore data.

meta

Optional (additional parameter)

Indicates the standard and custom metadata that can be specified during file upload. This parameter should be configured in the following format: key1:value1#key2:value2#key3:value3.

NOTE:
  1. The format example above indicates that the destination objects contain three groups of custom metadata: key1:value1, key2:value2, and key3:value3.
  2. Standard metadata headers include Content-Type, Content-Encoding, Cache-Control, Content-Disposition, Content-Language and Expires.

ps

Optional (additional parameter)

Indicates the size of each part in a multipart upload task, in bytes. The value ranges from 100KB to 5GB. The default value is the value of defaultPartSize in the configuration file.

NOTE:
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).
  • The parameter can be set to auto. In this case, obsutil automatically sets the part size for each multipart task based on the source file size.

cpd

Optional (additional parameter)

Indicates the folder where the part records reside. The default value is .obsutil_checkpoint, the subfolder in the home directory of the user who executes obsutil commands.

NOTE:

A part record is generated during a multipart upload and saved to the upload subfolder. After the upload succeeds, its part record is deleted automatically. If the upload fails or is suspended, the system attempts to resume the task according to its part record when you perform the upload the next time.

r

Mandatory for uploading a folder (additional parameter)

Optional for uploading multiple files/folders

Indicates files and subfolders within the folder when uploading a folder recursively.

f

Optional for uploading a folder or multiple files/folders (additional parameter)

Runs in force mode.

j

Optional for uploading a folder or multiple files/folders (additional parameter)

Indicates the maximum number of concurrent tasks for uploading a folder. The default value is the value of defaultJobs in the configuration file.

NOTE:

The value is ensured to be greater than or equal to 1.

msm

Mandatory for uploading multiple files/folders (additional parameter)

Enables the mode for uploading multiple files/folders. Possible values are 1 and 2.

NOTE:
  • If msm is set to 1, the source URL indicates a list of file/folder names separated by commas.
  • If msm is set to 2, the source URL indicates a file containing a list of file/folder names.
  • If the file or folder name already contains commas (,), do not set msm to 1.
  • If parameter r is not set, the folders in the list will not be uploaded.

exclude

Optional for uploading a folder or multiple files/folders (additional parameter)

Indicates the file matching patterns that are excluded, for example: *.txt.

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character. For instance, abc*.txt indicates any file whose name starts with abc and ends with .txt.
  • You can use \* to represent * and \? to represent ?.
  • If the name of the file to be uploaded matches the value of this parameter, the file is skipped.
NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and quotation marks for Windows.
  • The matching pattern applies to the absolute file path (including the file name and file directory).
  • The matching pattern takes effect only for files in the folder.
  • Multiple exclude parameters can be specified, for example, -exclude=*.xxx -exclude=*.xxx.

include

Optional for uploading a folder or multiple files/folders (additional parameter)

Indicates the file matching patterns that are included, for example: *.jpg.

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character.
  • You can use \* to represent * and \? to represent ?.
  • Only after identifying that the name of the file to be uploaded does not match the value of exclude, the system checks whether the file name matches the value of this parameter. If yes, the file is uploaded. If not, the file is skipped.
NOTE:

Example of uploading files in a request with the include parameter contained:

./obsutil cp /localpath/ obs://test/ -include=/localpath/2022-12-09/* -f -r

This command uploads files that are under localpath and start with /localpath/2022-12-09/ to bucket test.

NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and quotation marks for Windows.
  • The matching pattern applies to the absolute file path (including the file name and file directory).
  • The matching pattern takes effect only for files in the folder.
  • Multiple include parameters can be specified, for example, -include=*.xxx -include=*.xxx.

at

Optional for uploading a folder or multiple files/folders (additional parameter)

Indicates that only the files whose latest access time is within the value of timeRange are uploaded.

NOTE:
  • This parameter must be used together with timeRange.

disableDirObject

Optional for uploading multiple folders (additional parameter)

Indicates the folders themselves are not uploaded as an object. Configuring this parameter can avoid uploading empty folders to a bucket. If a folder contains files, the files will be uploaded and the original path format is retained.

timeRange

Optional for uploading a folder or multiple files/folders (additional parameter)

Indicates the time range matching pattern when uploading files. Only files whose latest modification time is within the configured time range are uploaded.

This pattern has a lower priority than the file matching patterns (exclude/include). That is, the time range matching pattern is executed after the configured file matching patterns.

NOTE:
  • The matching time range is represented in time1-time2, where time1 must be earlier than or the same as time2. The time format is yyyyMMddHHmmss.
  • Automatic formatting is supported. For example, yyyyMMdd is equivalent to yyyyMMdd000000, and yyyyMM is equivalent to yyyyMM01000000.
  • If this parameter is set to *-time2, all files whose latest modification time is earlier than time2 are matched. If it is set to time1-*, all files whose latest modification time is later than time1 are matched.
NOTICE:

Time in the matching pattern is the UTC time.

mf

Optional (additional parameter)

Indicates that the name matching pattern (include or exclude) and the time matching pattern (timeRange) also take effect on folders.

o

Optional (additional parameter)

Indicates the folder where operation result lists reside. After the command is executed, result lists (possibly including success, failure, and warning files) are generated in the folder. The default value is .obsutil_output, the subfolder in the home directory of the user who executes obsutil commands.

NOTE:
  • The naming rule for result lists is as follows: cp_{succeed | failed | warning}_report_time_TaskId.txt
  • By default, the maximum size of a single result list is 30 MB and the maximum number of result lists that can be retained is 1024. You can set the maximum size and number by configuring recordMaxLogSize and recordBackups in the configuration file.
  • If there are multiple folders and files and you need to confirm the detailed error information about a failed task, refer to the failure list cp_failed_report_time_TaskId.txt in the result list folder and the log files in the log path.

config

Optional (additional parameter)

User-defined configuration file for executing the current command. To learn the parameters that can be configured in this file, see Configuration Parameters.

e

Optional (additional parameter)

Specifies the endpoint.

i

Optional (additional parameter)

Specifies the user's AK.

k

Optional (additional parameter)

Specifies the user's SK.

t

Optional (additional parameter)

Specifies the user's security token.

Response

Field

Description

Parallel

Parameter -p in the request

Jobs

Parameter -j in the request

Threshold

Parameter -threshold in the request

PartSize

Parameter -ps in the request

Exclude

Parameter -exclude in the request

Include

Parameter -include in the request

TimeRange

Parameter -timeRange in the request

VerifyLength

Parameter -vlength in the request

VerifyMd5

Parameter -vmd5 in the request

CheckpointDir

Parameter -cpd in the request

OutputDir

Parameter -o in the request

ArcDir

Parameter -arcDir in the request

Succeed count

Number of successful tasks

Failed count

Number of failed tasks

Skip count

Number of tasks that are skipped during incremental upload, download, or copy, and synchronous upload, download, or copy.

NOTE:

Skipped tasks are recorded into successful tasks.

Warning count

Number of tasks that are executed successfully but contain warnings.

NOTE:
  • The task for which a warning is generated may be a failure or a success, which needs to be further determined according to the corresponding result list.
  • The number of tasks that generate warnings is independent of the number of successful or failed tasks. The total number of tasks is the number of successful tasks plus the number of failed tasks.

Succeed bytes

Number of bytes that are successfully uploaded or downloaded.

max cost

Maximum duration of all tasks, in ms

min cost

Minimum duration of all tasks, in ms

average cost

Average duration of all tasks, in ms

average tps

The average number of tasks completed per second

Task id

Unique ID of an operation, which is used to search for the result list generated in a batch task

Parent Topic: Upload