On this page

Show all

Help Center/ Object Storage Service/ SDK Reference/ PHP/ Object Upload/ Performing a Multipart Upload

Performing a Multipart Upload

Updated on 2024-04-30 GMT+08:00

View PDF

NOTICE:

If you have any questions during development, post them on the Issues page of GitHub. For details about parameters and usage of each API, see API Reference.

To upload a large file, multipart upload is recommended. Multipart upload is applicable to many scenarios, including:

Files to be uploaded are larger than 100 MB.
The network condition is poor. Connection to the OBS server is constantly down.
Sizes of files to be uploaded are uncertain.

Multipart upload consists of three phases:

Initialize a multipart upload (ObsClient->initiateMultipartUpload).
Upload parts one by one or concurrently (ObsClient->uploadPart).
Combine parts (ObsClient->completeMultipartUpload) or abort the multipart upload (ObsClient->abortMultipartUpload).

Initiating a Multipart Upload

Before using a multipart upload, you need to first let OBS initiate it. This operation will return an upload ID (globally unique identifier) created by the OBS server to identify the multipart upload. You can use this upload ID to initiate related operations, such as aborting a multipart upload, listing multipart uploads, and listing uploaded parts.

You can call ObsClient->initiateMultipartUpload to initialize a multipart upload.

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->initiateMultipartUpload([
       'Bucket' => 'bucketname',
       'Key' => 'objectname',
       'ContentType' => 'text/plain',
       'Metadata' => ['property' => 'property-value']
]);

printf("RequestId:%s\n",$resp['RequestId']);
printf("UploadId:%s\n",$resp['UploadId']);

NOTE:

When initializing a multipart upload, you can use the ContentType and Metadata parameters to respectively set the MIME type and customize the metadata of an object, besides the object name and owning bucket.
After the API for initializing a multipart upload is called, the upload ID will be returned. This ID will be used in follow-up operations.

Uploading a Part

After initializing a multipart upload, you can specify the object name and upload ID to upload a part. Each part has a part number (ranging from 1 to 10000). For parts with the same upload ID, their part numbers are unique and identify their comparative locations in the object. If you use the same part number to upload two parts, the latter one being uploaded will overwrite the former. The last part uploaded ranges from 0 to 5 GB in size, and each of the other parts ranges from 100 KB to 5 GB. Parts are uploaded in random order and can be uploaded through different processes or machines. OBS will combine them into the object based on their part numbers.

You can call ObsClient->uploadPart to upload a part.

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->uploadPart([
       'Bucket' => 'bucketname',
       'Key' => 'objectname',
       // Set the part number, which ranges from 1 to 10000.
       'PartNumber' => 1,
       // Set the upload ID.
       'UploadId' => 'upload id from initiateMultipartUpload',
// Set the large file to be uploaded. localfile is the path of the local file to be uploaded. You need to specify the file name.
       'SourceFile' => 'localfile',
       // Set the part size.
       'PartSize' => 5 * 1024 * 1024,
       // Set the start offset.
       'Offset' => 0
]);

printf("RequestId:%s\n",$resp['RequestId']);
printf("ETag:%s\n",$resp['ETag']);

NOTE:

Use the PartNumber parameter to specify the part number, the UploadId parameter to specify the globally unique ID, the SourceFile parameter to specify the to-be-uploaded file, the PartSize parameter to set the part size, and the Offset parameter to set the start offset of a part.
Except the part last uploaded, other parts must be larger than 100 KB. Part sizes will not be verified during upload because which one is last uploaded is not identified until parts are combined.
OBS will return ETags (MD5 values) of the received parts to users.
You can use the ContentMD5 parameter to set the MD5 value of the uploaded data.
Part numbers range from 1 to 10000. If the part number you set is out of this range, OBS will return error 400 Bad Request.
The minimum part size supported by an OBS 3.0 bucket is 100 KB, and the minimum part size supported by an OBS 2.0 bucket is 5 MB. You are advised to perform multipart upload to OBS 3.0 buckets.

Combining Parts

After all parts are uploaded, call the API for combining parts to generate the object. Before this operation, valid part numbers and ETags of all parts must be sent to OBS. After receiving this information, OBS verifies the validity of each part one by one. After all parts pass the verification, OBS combines these parts to form the final object.

You can call ObsClient->completeMultipartUpload to combine parts.

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->completeMultipartUpload([
       'Bucket' => 'bucketname',
       'Key' => 'objectname',
       // Set the upload ID.
       'UploadId' => 'upload id from initiateMultipartUpload',
       'Parts' => [
                     ['PartNumber' => 1, 'ETag' => 'etag value from uploadPart']
       ]
]);

printf("RequestId:%s\n",$resp['RequestId']);

CAUTION:

If the size of a part other than the last part is smaller than 100 KB, OBS returns 400 Bad Request.

NOTE:

Use the UploadId parameter to specify the globally unique identifier for the multipart upload and the Parts parameter to specify the list of part numbers and ETags. Content in the list is displayed in the ascending order by part number.
Part numbers can be inconsecutive.

Aborting a Multipart Upload

After a multipart upload is aborted, you cannot use its upload ID to perform any operation and the uploaded parts will be deleted by OBS.

When an object is being uploaded in multi-part mode or an object fails to be uploaded, parts generated in the bucket. These parts occupy your storage space. You can cancel the multi-part uploading task to delete unnecessary parts, thereby saving the storage space.

You can call ObsClient->abortMultipartUpload to abort a multipart upload.

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->abortMultipartUpload([
       'Bucket' => 'bucketname',
       'Key' => 'objectname',
       // Set the upload ID.
       'UploadId' => 'upload id from initiateMultipartUpload'
]);

printf("RequestId:%s\n",$resp['RequestId']);

Listing Uploaded Parts

You can call ObsClient->listParts to list successfully uploaded parts of a multipart upload.

The following table describes the parameters involved in this API.

Parameter	Description
UploadId	Upload ID, which globally identifies a multipart upload. The value is in the returned result of ObsClient->initiateMultipartUpload.
MaxParts	Maximum number of parts that can be listed per page.
PartNumberMarker	Part number after which listing uploaded parts begins. Only parts whose part numbers are larger than this value will be listed.

Listing parts in simple mode

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->listParts ( [ 
      'Bucket' => 'bucketname',
      'Key' => 'objectname',
      'UploadId' => 'upload id from initiateMultipartUpload' 
] );

printf ( "RequestId:%s\n", $resp ['RequestId'] );
foreach ( $resp ['Parts'] as $index => $part ) {
       printf ( "Parts[%d]\n", $index + 1 );
        // Part number, specified upon uploading
       printf ( "PartNumber:%s\n", $part ['PartNumber'] );
        // Time when the part was last uploaded
       printf ( "LastModified:%s\n", $part ['LastModified'] );
        // Part ETag
       printf ( "ETag:%s\n", $part ['ETag'] );
       // Part size
       printf ( "Size:%s\n", $part ['Size'] );
}

NOTE:

A maximum of 1,000 parts can be listed each time. If the upload of a specified ID contains more than 1,000 parts, IsTruncated in the response is true, indicating not all parts were listed. In such case, you can use NextPartNumberMarker to obtain the start position for next listing.
If you want to obtain all parts involved in a specific upload ID, you can use the paging mode for listing.

Listing all parts

If the number of parts of a multipart upload is larger than 1000, you can use the following sample code to list all parts.

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$partNumberMarker = null;
$index = 1;
do{
       $resp = $obsClient->listParts ( [ 
             'Bucket' => 'bucketname',
             'Key' => 'objectname',
             'UploadId' => 'upload id from initiateMultipartUpload',
             'PartNumberMarker' => $partNumberMarker
       ] );
       
       printf ( "RequestId:%s\n", $resp ['RequestId'] );
       foreach ( $resp ['Parts'] as $part ) {
              printf ( "Parts[%d]\n", $index );
              // Part number, specified upon uploading
              printf ( "PartNumber:%s\n", $part ['PartNumber'] );
              // Time when the part was last uploaded
              printf ( "LastModified:%s\n", $part ['LastModified'] );
              // Part ETag
              printf ( "ETag:%s\n", $part ['ETag'] );
              // Part size
              printf ( "Size:%s\n", $part ['Size'] );
              $index ++;
       }
       
       $partNumberMarker = $resp['NextPartNumberMarker'];
       
}while($resp['IsTruncated']);

Listing Multipart Uploads

You can call ObsClient->listMultipartUploads to list multipart uploads. The following table describes related parameters.

Parameter	Description
Prefix	Prefix that the object names in the multipart uploads to be listed must contain
Delimiter	Character used to group object names involved in multipart uploads. If the object name contains the Delimiter parameter, the character string from the first character to the first delimiter in the object name is grouped under a single result element, CommonPrefix. (If a prefix is specified in the request, the prefix must be removed from the object name.)
MaxUploads	Maximum number of listed multipart uploads. The value ranges from 1 to 1000. If the value is not in this range, 1000 parts are listed by default.
KeyMarker	Object name to start with when listing multipart uploads
UploadIdMarker	Upload ID after which the multipart upload listing begins. It is effective only when used with KeyMarker so that multipart uploads after UploadIdMarker of KeyMarker will be listed.

Listing multipart uploads in simple mode

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$resp = $obsClient->listMultipartUploads ( [ 
      'Bucket' => 'bucketname' 
] );

printf ( "RequestId:%s\n", $resp ['RequestId'] );
foreach ( $resp ['Uploads'] as $index => $upload ) {
       printf ( "Uploads[%d]\n", $index + 1 );
       printf ( "Key:%s\n", $upload ['Key'] );
       printf ( "UploadId:%s\n", $upload ['UploadId'] );
       printf ( "Initiated:%s\n", $upload ['Initiated'] );
       printf ( "Owner[ID]:%s\n", $upload ['Owner'] ['ID'] );
       printf ( "StorageClass:%s\n", $upload ['StorageClass'] );
}

NOTE:

Information about a maximum of 1000 multipart uploads can be listed each time. If a bucket contains more than 1000 multipart uploads and IsTruncated is true in the returned result, not all uploads are listed. In such cases, you can use NextKeyMarker and NextUploadIdMarker to obtain the start position for next listing.
If you want to obtain all multipart uploads in a bucket, you can list them in paging mode.

Listing all multipart uploads

// Import the dependency library.
require 'vendor/autoload.php';
// Import the SDK code library during source code installation.
// require 'obs-autoloader.php';
// Declare the namespace.
use Obs\ObsClient;
// Create an instance of ObsClient.
$obsClient = new ObsClient ( [ 
      //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
      //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
      'key' => getenv('ACCESS_KEY_ID'),
      'secret' => getenv('SECRET_ACCESS_KEY'),
      'endpoint' => 'https://your-endpoint'
] );

$keyMarker = null;
$uploadIdMarker = null;
$index = 1;
do{
       $resp = $obsClient->listMultipartUploads ( [ 
             'Bucket' => 'bucketname',
             'KeyMarker' => $keyMarker,
             'UploadIdMarker' => $uploadIdMarker
       ] );
       
       printf ( "RequestId:%s\n", $resp ['RequestId'] );
       foreach ( $resp ['Uploads'] as $index => $upload ) {
              printf ( "Uploads[%d]\n", $index );
              printf ( "Key:%s\n", $upload ['Key'] );
              printf ( "UploadId:%s\n", $upload ['UploadId'] );
              printf ( "Initiated:%s\n", $upload ['Initiated'] );
              printf ( "Owner[ID]:%s\n", $upload ['Owner'] ['ID'] );
              printf ( "StorageClass:%s\n", $upload ['StorageClass'] );
              $index ++;
       }
       $keyMarker = $resp['NextKeyMarker'];
       $uploadIdMarker = $resp['NextUploadIdMarker'];
}while ($resp['IsTruncated']);

Parent topic: Object Upload

Previous topic: Setting Object Properties

Next topic: Performing a Multipart Copy

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot