Performing a Multipart Upload

Initiating a Multipart Upload

Before using a multipart upload, you need to first initiate it. This operation will return an upload ID (globally unique identifier) created by the OBS server to identify the multipart upload. You can use this upload ID in your subsequent requests including abortMultipartUpload, listMultipartUploads, and listParts.

You can call ObsClient.initiateMultipartUpload to initiate a multipart upload.

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

obsClient.initiateMultipartUpload({
       Bucket : 'bucketname',
       Key : 'objectname',
       ContentType : 'text/plain',
       Metadata : {'property' : 'property-value'}
}, function (err, result) {
       if(err){
              console.error('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     console.log('UploadId-->' + result.InterfaceResult.UploadId);
              }
       }
});

• When initiating a multipart upload, you can use the ContentType and Metadata parameters to respectively set the MIME type and custom metadata of an object.
• After the API for initiating a multipart upload is successfully called, an upload ID will be returned. This ID will be used in subsequent operations.

Uploading Parts

After initiating a multipart upload, you can specify the object name and upload ID to upload a part. Each part has a part number (ranging from 1 to 10,000). For parts with the same upload ID, their part numbers are unique and identify their relative location in the object. If you use the same part number to upload two parts, the latter one uploaded will overwrite the former one. The last part uploaded can be up to 5 GB in size, and the size of each of the other parts is in the range of 100 KB to 5 GB. Parts can be uploaded in random order, or even through different processes or machines. OBS will combine them into the final object based on their part numbers.

You can call ObsClient.uploadPart to upload parts.

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

const bucketname = 'examplebucket';
const objectname = 'exampleobject';
const PartSize = 5 * 1024 * 1024;
const Uploadid = 'upload id from initiateMultipartUpload';
const file = document.getElementById('input-file').files[0];
const lastPartSize = file.size % PartSize;
// Number of parts
const count = Math.ceil(file.size / PartSize);
// Upload part n.
const uploadPart = (n) => {
    obsClient.uploadPart({
        Bucket: bucketname,
        Key: objectname,
       // Set the part number, which ranges from 1 to 10000.
        PartNumber: n,
       // Set the upload ID.
        UploadId,
       // Specify the large file to be uploaded.
        SourceFile: file,
       // Set the part size.
        PartSize: count === n ? lastPartSize : PartSize,
       // Set the start offset.
        Offset: (n-1) * PartSize
    }, function (err, result) {
        if(err){
                console.log('Error-->' + err);
        }else{
                console.log('Status-->' + result.CommonMsg.Status);
                if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                      console.log('ETag-->' + result.InterfaceResult.ETag);
                }
        }
    });
}

// Upload part 1.
uploadPart(1);

If the ETag value obtained is undefined, you need to configure a CORS rule and add the ETag to the additional header. For details, see ETag.

• Use the PartNumber parameter to specify the part number, the UploadId parameter to specify the globally unique ID, the SourceFile parameter to specify the to-be-uploaded file, the PartSize parameter to set the part size, and the Offset parameter to set the start offset of the file.
• SourceFile must indicate a File or Blob object. For example, on an HTML page, use an input tag whose type is file to specify the to-be-uploaded file: <input type="file" id="input-file"/>.
• Except the part last uploaded, other parts must be larger than 100 KB. The size of a part is not verified during its upload, because the system cannot define whether it is the last part. The part size is verified when the parts are assembled.
• OBS will return ETags (MD5 values) of the received parts to users.
• You can use the ContentMD5 parameter to set the MD5 value of the uploaded data.
• Part numbers range from 1 to 10000. If a part number exceeds this range, OBS will return error 400 Bad Request.
• The minimum part size supported by an OBS 3.0 bucket is 100 KB, and that supported by an OBS 2.0 bucket is 5 MB. You are advised to perform multipart uploads on OBS 3.0 buckets.

Assembling Parts

After all parts are uploaded, call the API for assembling parts to generate the object. Before this operation, valid part numbers and ETags of all parts must be sent to OBS. After receiving this information, OBS verifies the validity of each part one by one. After all parts pass the verification, OBS assembles these parts to form the final object.

You can call ObsClient.completeMultipartUpload to assemble parts.

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

obsClient.completeMultipartUpload({
       Bucket:'bucketname',
       Key:'objectname',
       // Set the upload ID.
       UploadId:'upload id from initiateMultipartUpload',
       Parts: [{'PartNumber':1,'ETag':'etag value from uploadPart'}]
}, function (err, result) {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
       }
});

• If the size of a part other than the last part is smaller than 100 KB, OBS returns 400 Bad Request.

• Use the UploadId parameter to specify the globally unique identifier for the multipart upload and the Parts parameter to specify the list of part numbers and ETags. Content in the list is displayed in the ascending order by part number.
• Part numbers can be inconsecutive.

Aborting a Multipart Upload

After a multipart upload is aborted, you cannot use its upload ID to perform any operation and the uploaded parts will be deleted by OBS.

When an object is being uploaded or fails to be uploaded in multipart mode, parts are generated in the bucket. These parts occupy your storage space. You can abort the multipart upload to delete unnecessary parts, thereby saving the storage space.

You can call ObsClient.abortMultipartUpload to abort a multipart upload.

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

obsClient.abortMultipartUpload({
       Bucket:'bucketname',
       Key:'objectname',
       // Set the upload ID.
       UploadId:'upload id from initiateMultipartUpload',
}, function (err, result) {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
       }
});

Listing Uploaded Parts

You can call ObsClient.listParts to list successfully uploaded parts of a multipart upload.

The following table describes the parameters involved in this API.

• Listing parts in simple mode

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

// List uploaded parts. uploadId is obtained from initiateMultipartUpload.
obsClient.listParts({
       Bucket : 'bucketname',
       Key: 'objectname',
       UploadId : 'upload id from initiateMultipartUpload'
}, function (err, result) {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     for(var i in result.InterfaceResult.Parts){
                           console.log('Part['+ i +']:');
                           // Part number, specified during the upload
                           console.log('PartNumber-->' + result.InterfaceResult.Parts[i]['PartNumber']);
                           // Time when the part was last uploaded
                           console.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']);
                           // Part ETag
                           console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']);
                           // Part size
                           console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']);
                     }
              }
       }
});

• A maximum of 1,000 parts can be listed each time. If an upload identified by a specified ID contains more than 1,000 parts, the value of InterfaceResult.IsTruncated in the response is true, indicating that not all parts are listed. In such cases, you can obtain the start position for the next listing through InterfaceResult.NextPartNumberMarker.
• If you want to obtain all parts involved in a specific upload ID, you can use the paging mode for listing.

• Listing all parts

The following example lists all parts of a multipart upload when the number of parts exceeds 1,000.

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

var listAll = function (partNumberMarker) {
       // List uploaded parts. uploadId is obtained from initiateMultipartUpload.
       obsClient.listParts({
              Bucket : 'bucketname',
              Key: 'objectname',
              UploadId : 'upload id from initiateMultipartUpload',
              PartNumberMarker : partNumberMarker
       }, function (err, result) {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(var i in result.InterfaceResult.Parts){
                                  console.log('Part['+ i +']:');
                                  // Part number, specified during the upload
                                  console.log('PartNumber-->' + result.InterfaceResult.Parts[i]['PartNumber']);
                                  // Time when the part was last uploaded
                                  console.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']);
                                  // Part ETag
                                  console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']);
                                  // Part size
                                  console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']);
                           }
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextPartNumberMarker);
                           }
                     }
              }
       });
};

listAll();

List Multipart Uploads

You can call ObsClient.listMultipartUploads to list multipart uploads. The following table describes the parameters involved in this API.

• Listing multipart uploads in simple mode

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

obsClient.listMultipartUploads({
       Bucket : 'bucketname'
}, function (err, result) {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     for(var i in result.InterfaceResult.Uploads){
                           console.log('Uploads[' + i + ']');
                           console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']);
                           console.log('Key-->' + result.InterfaceResult.Uploads[i]['Key']);
                           console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']);
                     }
              }
       }
});

• A maximum of 1,000 multipart uploads can be listed each time. If a bucket contains more than 1,000 multipart uploads, InterfaceResult.IsTruncated in the response is true, indicating not all uploads were listed. In such case, you can use InterfaceResult.NextKeyMarker and InterfaceResult.NextUploadIdMarker to obtain the start position for the next listing.
• If you want to obtain all multipart uploads in a bucket, you can list them in paging mode.

• Listing all multipart uploads

// Create an ObsClient instance.
var obsClient = new ObsClient({
    // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
    // The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    access_key_id: process.env.AccessKeyID,
    secret_access_key: process.env.SecretAccessKey,
    // CN-Hong Kong is used here in this example. Replace it with the one currently in use.
    server: 'https://obs.ap-southeast-1.myhuaweicloud.com'
});

var listAll = function (keyMarker, uploadIdMarker) {
       obsClient.listMultipartUploads({
              Bucket : 'bucketname',
              KeyMarker : keyMarker,
              UploadIdMarker : uploadIdMarker
       }, function (err, result) {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(var i in result.InterfaceResult.Uploads){
                                  console.log('Uploads[' + i + ']');
                                  console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']);
                                  console.log('Key-->' + result.InterfaceResult.Uploads[i]['Key']);
                                  console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']);
                           }
                           
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextKeyMarker, result.InterfaceResult.NextUploadIdMarker);
                           }
                     }
              }
       });
}

listAll();