Help Center/ Object Storage Service/ SDK Reference/ Node.js/ Object Upload (SDK for Node.js)/ Uploading an Object - Resumable (SDK for Node.js)
Updated on 2024-11-13 GMT+08:00
View PDF

Uploading an Object - Resumable (SDK for Node.js)

If you have any questions during development, post them on the Issues page of GitHub.

Function

The resumable upload is an encapsulated and enhanced version of the multipart upload used for dealing with possible upload failures of large files when the network connection is unstable or a program crashes.

Uploading large files often fails due to poor network conditions or program breakdowns. It is a waste of resources to restart the upload process upon an upload failure, and the restarted upload process may still suffer from the unstable network. To resolve such issues, you can use the API for resumable upload, whose working principle is to divide the to-be-uploaded file into multiple parts and upload them separately. The upload result of each part is recorded in a checkpoint file in real time. Only when all parts are successfully uploaded, the result indicating a successful upload will be returned. Otherwise, an error is returned in callback function to remind you of calling the API again for re-uploading. Based on the upload status of each part recorded in the checkpoint file, the re-uploading will upload the parts failed to be uploaded previously, instead of uploading all parts. By virtue of this, resources are saved and efficiency is improved.

The resumable upload interface helps save resources and improve efficiency by restarting an upload from the point of failure and concurrently uploading parts. You do not need to worry about internal service details, such as the creation and deletion of checkpoint files, division of objects, or concurrent uploads of parts.

Restrictions

The total size of files uploaded by the resumable upload API must be larger than 100 KB.

Method

ObsClient.uploadFile(params)

Request Parameters

Table 1 List of request parameters

Parameter

Type

Mandatory (Yes/No)

Description

Bucket

string

Yes

Explanation:

Bucket name.

Restrictions:

  • A bucket name must be unique across all accounts and regions.
  • A bucket name:
    • Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed.
    • Cannot be formatted as an IP address.
    • Cannot start or end with a hyphen (-) or period (.).
    • Cannot contain two consecutive periods (..), for example, my..bucket.
    • Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket.
  • If you repeatedly create buckets with the same name in the same region, no error will be reported, and the bucket attributes comply with those set in the first creation request.

Value range:

The value can contain 3 to 63 characters.

Default value:

None

Key

string

Yes

Explanation:

Object name. An object is uniquely identified by an object name in a bucket. An object name is a complete path of the object that does not contain the bucket name.

For example, if the address for accessing the object is examplebucket.obs.eu-west-101.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt.

Restrictions:

None

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

UploadFile

string

No

Explanation:

Source file path of the object to be uploaded

Restrictions:

None

Value range:

None

Default value:

None

PartSize

number

No

Explanation:

Size of the current part.

Restrictions:

None

Value range:

The value ranges from 100 KB to 5 GB, in bytes.

Default value:

102400

TaskNum

number

No

Explanation:

Maximum number of parts that can be uploaded concurrently

Restrictions:

None

Value range:

1~10000

Default value:

1, indicating concurrent uploads are not used

ProgressCallback

function

No

Explanation:

Callback function for obtaining the upload progress

NOTE:

This callback function contains the following parameters in sequence: number of uploaded bytes, total bytes, and used time (in seconds).

Restrictions:

None

Value range:

None

Default value:

None

ResumeCallback

function

No

Explanation:

Callback function used to obtain the control parameter for canceling a resumable download

Restrictions:

None

Value range:

None

Default value:

None

NOTE:
  • This callback function contains a control parameter used for pausing or aborting a resumable upload.
  • By calling the cancel method of this control parameter, you can pause a resumable upload.
  • By calling the abort method of this control parameter, you can abort a resumable upload.

EnableCheckpoint

boolean

No

Explanation:

Whether to enable the resumable mode.

Restrictions:

None

Value range:

  • true: The resumable mode is enabled.
  • false: The resumable mode is disabled. In this case, this API works as a multipart upload API, and no checkpoint files are generated.

Default value:

false

CheckpointFile

string

No

Explanation:

Path of a file generated for recording the progress of a resumable upload. The file contains the information about parts and progress.

Restrictions:

This parameter is valid only in the resumable mode.

Value range:

None

Default value:

If this parameter is left blank, the checkpoint file will be in the same directory as the local file to be uploaded.

EnableCheckSum

boolean

No

Explanation:

Whether to verify the file to upload. If this function is enabled, before each task restarts, the system verifies whether the file to upload is the one used during task initialization.

Restrictions:

This parameter is valid only in the resumable mode.

Value range:

  • true: The file is verified.
  • false: The file is not verified.

Default value:

false

ContentType

string

No

Explanation:

MIME type of the file to be uploaded. MIME type is a standard way of describing a data type and is used by the browser to decide how to display data.

Restrictions:

None

Value range:

See What Is Content-Type (MIME)? (SDK for Node.js)

Default value:

If you do not specify this parameter when uploading an object, the SDK determines the object type based on the suffix of the specified object name and automatically assigns a value to this parameter.

ACL

AclType

No

Explanation:

Access control list (ACL) that can be pre-defined when an object is created. For details about ACLs, see ACLs.

Restrictions:

None

Value range:

See Table 2.

Default value:

None

WebsiteRedirectLocation

string

No

Explanation:

If the bucket is configured with website hosting, the request for obtaining the object can be redirected to another object in the bucket or an external URL.

The request is redirected to object anotherPage.html in the same bucket:

WebsiteRedirectLocation:/anotherPage.html

The request is redirected to an external URL:

WebsiteRedirectLocation:http://www.example.com/

OBS obtains the specified value from the header and stores it in the object metadata WebsiteRedirectLocation.

Restrictions:

  • The value must start with a slash (/), http://, or https:// and cannot exceed 2 KB.
  • OBS only supports redirection of objects that are in the root directory.

Value range:

None

Default value:

None

SseKmsKey

string

No

Explanation:

ID of the KMS master key when SSE-KMS is used.

Restrictions:

Valid value formats are as follows:

  1. regionID:domainID:key/key_id
  2. key_id

In the preceding formats:

Value range:

None

Default value:

  • If this parameter is not specified, the default master key will be used.
  • If there is not a default master key, OBS will create one and use it.

SseC

string

Yes when SSE-C is used

Explanation:

SSE-C is used for encrypting objects on the server side.

Restrictions:

Only AES256 is supported.

Value range:

AES256

Default value:

None

SseCKey

string

Yes when SSE-C is used

Explanation:

Key used for encrypting the object when SSE-C is used

Restrictions:

A Base64-encoded, 256-bit value, for example, K7QkYpBkM5+hca27fsNkUnNVaobncnLht/rCB2o/9Cw=

Value range:

None

Default value:

None

SseCKey

string

Yes if used as a request parameter

Explanation:

Key used for encrypting the object when SSE-C is used

Restrictions:

A Base64-encoded, 256-bit value, for example, K7QkYpBkM5+hca27fsNkUnNVaobncnLht/rCB2o/9Cw=

Value range:

None

Default value:

None

Metadata

object

No

Explanation:

Custom metadata of the object to be uploaded. You can add a header starting with x-obs-meta- in the request to define metadata. The custom metadata will be returned in the response when you retrieve the object or query the object metadata.

Restrictions:

  • The custom metadata cannot exceed 8 KB in total. To measure the size, calculate the sum of bytes of all UTF-8 encoded keys and values.
  • The custom metadata keys are case insensitive, but are stored in lowercase in OBS. The key values are case sensitive.
  • Both custom metadata keys and their values must conform to US-ASCII standards. If non-ASCII or unrecognizable characters are required, they must be encoded and decoded in URL or Base64 on the client, because the server does not perform such operations.

Value range:

None

Default value:

None
Table 2 AclType

Constant

Default Value

Description

ObsClient.enums.AclPrivate

private

Private read and write

A bucket or object can only be accessed by its owner.

ObsClient.enums.AclPublicRead

public-read

Public read and private write

If this permission is granted on a bucket, anyone can read the object list, multipart uploads, metadata, and object versions in the bucket.

If this permission is granted on an object, everyone can obtain the content and metadata of the object.

ObsClient.enums.AclPublicReadWrite

public-read-write

Public read and write

If this permission is granted on a bucket, anyone can read the object list, multipart tasks, metadata, and object versions in the bucket and can upload or delete objects, initiate multipart upload tasks, upload parts, assemble parts, copy parts, and abort multipart upload tasks.

If this permission is granted on an object, everyone can obtain the content and metadata of the object.

ObsClient.enums.AclPublicReadDelivered

public-read-delivered

Public read on a bucket as well as the objects in the bucket.

If this permission is granted on a bucket, anyone can read the object list, multipart tasks, metadata, and object versions and read the content and metadata of objects in the bucket.

NOTE:

AclPublicReadDelivered does not apply to objects.

ObsClient.enums.AclPublicReadWriteDelivered

public-read-write-delivered

Public read and write on a bucket as well as the objects in the bucket.

If this permission is granted on a bucket, anyone can read the object list, multipart uploads, metadata, and object versions in the bucket and can upload or delete objects, initiate multipart upload tasks, upload parts, assemble parts, copy parts, and abort multipart uploads. They can also read the content and metadata of objects in the bucket.

NOTE:

AclPublicReadWriteDelivered does not apply to objects.

ObsClient.enums.AclBucketOwnerFullControl

bucket-owner-full-control

If this permission is granted on an object, only the bucket and object owners have the full control over the object.

By default, if you upload an object to a bucket of any other user, the bucket owner does not have the permissions on your object. After you grant this policy to the bucket owner, the bucket owner can have full control over your object. For example, if user A uploads object x to user B's bucket, user B does not have the control over object x. If user A sets the bucket-owner-full-control policy for object x, user B then has the control over object x.

Responses

Table 3 Responses

Type

Description

Table 4

NOTE:

This API returns a Promise response, which requires the Promise or async/await syntax.

Explanation:

Returned results. For details, see Table 4.
Table 4 Response

Parameter

Type

Description

CommonMsg

ICommonMsg

Explanation:

Common information generated after an API call is complete, including the HTTP status code and error code. For details, see Table 5.

InterfaceResult

Table 6

Explanation:

Results outputted for a successful call. For details, see Table 6.

Restrictions:

This parameter is not included if the value of Status is greater than 300.
Table 5 ICommonMsg

Parameter

Type

Description

Status

number

Explanation:

HTTP status code returned by the OBS server.

Value range:

A status code is a group of digits indicating the status of a response. It ranges from 2xx (indicating successes) to 4xx or 5xx (indicating errors). For details, see Status Codes.

Code

string

Explanation:

Error code returned by the OBS server.

Message

string

Explanation:

Error description returned by the OBS server.

HostId

string

Explanation:

Request server ID returned by the OBS server.

RequestId

string

Explanation:

Request ID returned by the OBS server.

Id2

string

Explanation:

Request ID2 returned by the OBS server.

Indicator

string

Explanation:

Error code details returned by the OBS server.
Table 6 UploadFileOutput

Parameter

Type

Description

RequestId

string

Explanation:

Request ID returned by the OBS server

ETag

string

Explanation:

Base64-encoded, 128-bit MD5 value of an assembled object calculated based on the ETag of each part. The ETag is a unique identifier for the object's contents, used to determine if the object has been updated. For example, if the ETag value is A when an object is uploaded and B when the object is downloaded, this indicates the contents of the object have been changed. The ETag reflects changes to an object's contents, not its metadata. Objects created by the upload and copy operations have unique ETags after being encrypted using MD5.

Restrictions:

If an object is encrypted using server-side encryption, the ETag is not the MD5 value of the object.

Bucket

string

Explanation:

Name of the bucket in which parts are assembled

Key

string

Explanation:

Object name. An object is uniquely identified by an object name in a bucket. An object name is a complete path of the object that does not contain the bucket name.

For example, if the address for accessing the object is examplebucket.obs.eu-west-101.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt.

Location

string

Explanation:

URL of the object obtained from assembling the parts.

Example: https://example-Bucket.obs.regions.myhuaweicloud.com/example-Object

VersionId

string

Explanation:

Version ID of the object obtained from assembling the parts. If versioning is enabled for the bucket, the object version ID will be returned.

SseKms

string

Explanation:

SSE-KMS is used for encrypting objects on the server side.

SseKmsKey

string

Explanation:

ID of the KMS master key when SSE-KMS is used.

SseC

string

Explanation:

SSE-C is used for encrypting objects on the server side.

SseCKeyMd5

string

Explanation:

MD5 value of the key used for encrypting objects when SSE-C is used. This value is used to check whether any error occurs during the transmission of the key.

Restrictions:

Base64-encoded MD5 value of the key, for example, 4XvB3tbNTN+tIEVa0/fGaQ==

Code Examples

This example uploads example/objectname to examplebucket using resumable upload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
// Import the OBS library.
// Use npm to install the client.
const ObsClient = require("esdk-obs-nodejs");
// Use the source code to install the client.
// var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
const obsClient = new ObsClient({
  // Obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways. Using hard coding may result in leakage.
  // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/eu/usermanual-ca/ca_01_0003.html.
  access_key_id: process.env.ACCESS_KEY_ID,
  secret_access_key: process.env.SECRET_ACCESS_KEY,
  // (Optional) If you use a temporary AK/SK pair and a security token to access OBS, you are advised not to use hard coding, which may result in information leakage. You can obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways.
  // security_token: process.env.SECURITY_TOKEN,
  // Enter the endpoint corresponding to the region where the bucket is located. EU-Dublin is used here in this example. Replace it with the one currently in use.
  server: "https://obs.eu-west-101.myhuaweicloud.eu"
});

async function uploadFile() {
  try {
    const params = {
      // Specify the bucket name.
      Bucket: 'examplebucket',
      // Specify the object to be created (example/objectname in this example).
      Key: 'example/objectname',
      //Specify the local file to upload (/tmp/objectname in this example).
      UploadFile: 'localfile',
      // Specify whether to enable resumable transmission. Value true is used in this example. The default value is false.
      EnableCheckpoint: true,
      // Specify a part size, in bytes. This example sets each part to 9 MB.
      PartSize: 9 * 1024 * 1024,
      // Specify the maximum number of parts that can be concurrently transmitted. 5 is used in this example.
      TaskNum: 5
    };
    // Upload the object using resumable upload.
    const result = await obsClient.uploadFile(params);
    if (result.CommonMsg.Status <= 300) {
      console.log("Upload file(%s) under the bucket(%s) successful!", params.Key, params.Bucket);
      console.log("RequestId: %s", result.CommonMsg.RequestId);
      fmt.Printf("ETag:%s\n", result.InterfaceResult.ETag)
      return;
    };
    console.log("An ObsError was found, which means your request sent to OBS was rejected with an error response.");
    console.log("Status: %d", result.CommonMsg.Status);
    console.log("Code: %s", result.CommonMsg.Code);
    console.log("Message: %s", result.CommonMsg.Message);
    console.log("RequestId: %s", result.CommonMsg.RequestId);
  } catch (error) {
    console.log("An Exception was found, which means the client encountered an internal problem when attempting to communicate with OBS, for example, the client was unable to access the network.");
    console.log(error);
  };
};

uploadFile();