Downloading an Object - Resumable (SDK for Go)
Function
Downloading large files often fails due to poor network conditions or program breakdowns. It is a waste of resources to restart the download process upon a download failure, and the restarted download process may still suffer from the unstable network. To resolve such issues, the resumable download API splits the file to be downloaded into multiple parts and downloads them separately. The download result of each part is recorded in a checkpoint file in real time. Only when all parts are successfully downloaded, the result indicating a successful download is returned. Otherwise, an error message is returned to remind you of calling the API again for re-downloading. Based on the download status of each part recorded in the checkpoint file, the re-downloading will download the parts failed to be downloaded previously, instead of downloading all parts. By virtue of this, resources are saved and efficiency is improved.
This API calls ObsClient.DownloadFile to perform a resumable download.
Restrictions
- To download an object, you must be the bucket owner or have the required permission (obs:object:GetObject in IAM or GetObject in a bucket policy). For details, see Introduction to OBS Access Control, IAM Custom Policies, and Configuring an Object Policy.
- The mapping between OBS regions and endpoints must comply with what is listed in Regions and Endpoints.
- The resumable download API, which is implemented based on partial download, is an encapsulated and enhanced version of partial download.
- This API saves resources and improves efficiency upon the re-download, and speeds up the download process by concurrently downloading parts. Because this API is invisible to users, users are unaware of internal service details, such as the creation and deletion of checkpoint files, division of objects, and concurrent download of parts.
- EnableCheckpoint: The default value is False, indicating that resumable download is disabled. In this case, the resumable download API is a simple encapsulation of the partial download API, and no checkpoint file will be generated.
- CheckpointFile: This parameter is effective only when EnableCheckpoint is True.
Method
func (obsClient ObsClient) DownloadFile(input *DownloadFileInput) (output *GetObjectMetadataOutput, err error)
Request Parameters
Parameter |
Type |
Mandatory (Yes/No) |
Description |
---|---|---|---|
input |
Yes |
Explanation: Input parameters for a resumable download. For details, see Table 2. |
Parameter |
Type |
Mandatory (Yes/No) |
Description |
---|---|---|---|
Bucket |
string |
Yes |
Explanation: Bucket name Restrictions:
Default value: None |
Key |
string |
Yes |
Explanation: Object name. An object is uniquely identified by an object name in a bucket. An object name is a complete path that does not contain the bucket name. For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt. Value range: The value must contain 1 to 1,024 characters. Default value: None |
VersionId |
string |
No |
Explanation: Object version ID, for example, G001117FCE89978B0000401205D5DC9 Value range: The value must contain 32 characters. Default value: None. If this parameter is left blank, the latest version of the object is obtained. |
DownloadFile |
string |
No |
Explanation: Full path of the local directory to which the object is downloaded. Default value: If the parameter is left blank, the downloaded object is saved in the directory where the program is executed. |
PartSize |
int64 |
No |
Explanation: Part size Value range: The value must be greater than 0 but less than the object size, in bytes. Default value: 9437184, indicating 9 MB for each part |
TaskNum |
int |
No |
Explanation: Maximum number of parts that can be downloaded concurrently Value range: The value must be greater than 0 but not exceed the result of the file size divided by the part size (rounded up). Default value: 1, indicating concurrent downloads are not used. |
EnableCheckpoint |
bool |
No |
Explanation: Whether to enable the resumable download Value range:
Default value: false |
CheckpointFile |
string |
No |
Explanation: File used to record the download progress. This parameter is valid only in a resumable download. Default value: If the parameter is left blank, the file is in the same local directory as the downloaded object. |
SseHeader |
No |
Explanation: Server-side decryption headers. For details, see Table 3. |
|
IfMatch |
string |
No |
Explanation: Preset ETag. If the ETag of the object to be downloaded is the same as the preset ETag, the object is returned. Otherwise, an error is returned. Value range: The value must contain 32 characters. Default value: None |
IfNoneMatch |
string |
No |
Explanation: Preset ETag. If the ETag of the object to be downloaded is different from the preset ETag, the object is returned. Otherwise, an error is returned. Value range: The value must contain 32 characters. Default value: None |
IfModifiedSince |
time.Time |
No |
Explanation: The object is returned if it has been modified since the specified time; otherwise, an error is returned. Restrictions: The time must be in the ISO8601 format, for example, 2018-01-01T00:00:00.000Z. Sample code: time.Now().Add(time.Duration(24) * time.Hour) Default value: None |
IfUnmodifiedSince |
time.Time |
No |
Explanation: The object is returned if it has not been modified since the specified time; otherwise, an error is returned. Restrictions: The time must be in the ISO8601 format, for example, 2018-01-01T00:00:00.000Z. Sample code: time.Now().Add(time.Duration(24) * time.Hour) Default value: None |
Parameter |
Type |
Mandatory (Yes/No) |
Description |
---|---|---|---|
Encryption |
string |
Yes if used as a request parameter |
Explanation: SSE-C used for encrypting objects Value range: AES256, indicating objects are encrypted using SSE-C Default value: None |
Key |
string |
Yes if used as a request parameter |
Explanation: Key for encrypting the object when SSE-C is used Restrictions: The value is a Base64-encoded 256-bit key, for example, K7QkYpBkM5+hca27fsNkUnNVaobncnLht/rCB2o/9Cw=. Default value: None |
KeyMD5 |
string |
No if used as a request parameter |
Explanation: MD5 value of the key for encrypting objects when SSE-C is used. This value is used to check whether any error occurs during the transmission of the key. Restrictions: The value is encrypted by MD5 and then encoded by Base64, for example, 4XvB3tbNTN+tIEVa0/fGaQ==. Default value: None |
Responses
Parameter |
Type |
Description |
---|---|---|
output |
Explanation: Returned results. For details, see Table 5. |
|
err |
error |
Explanation: Error messages returned by the API |
Parameter |
Type |
Description |
---|---|---|
StatusCode |
int |
Explanation: HTTP status code Value range: A status code is a group of digits that can be 2xx (indicating successes) or 4xx or 5xx (indicating errors). It indicates the status of a response. For more information, see Status Code. Default value: None |
RequestId |
string |
Explanation: Request ID returned by the OBS server Default value: None |
ResponseHeaders |
map[string][]string |
Explanation: HTTP response headers Default value: None |
StorageClass |
Explanation: Object storage class Value range: See Table 6. Default value: None |
|
AllowOrigin |
string |
Explanation: If Origin in the request meets the CORS rules of the bucket, AllowedOrigin specified in the CORS rules is returned. AllowedOrigin indicates the origin from which the requests can access the bucket. Restrictions: Domain name of the origin. Each origin can contain only one wildcard character (*), for example, https://*.vbs.example.com. Default value: None |
AllowHeader |
string |
Explanation: If RequestHeader in the request meets the CORS rules of the bucket, AllowedHeader specified in the CORS rules is returned. AllowedHeader indicates the allowed headers for cross-origin requests. Only CORS requests matching the allowed headers are valid. Restrictions: Each header can contain only one wildcard character (*). Spaces, ampersands (&), colons (:), and less-than signs (<) are not allowed. Default value: None |
AllowMethod |
string |
Explanation: AllowedMethod in the CORS rules of the bucket. It specifies the HTTP method of cross-origin requests, that is, the operation type of buckets and objects. Value range: The following HTTP methods are supported:
Default value: None |
ExposeHeader |
string |
Explanation: ExposeHeader in the CORS rules of the bucket. It specifies the CORS-allowed additional headers in the response. These headers provide additional information to clients. By default, your browser can only access headers Content-Length and Content-Type. If your browser needs to access other headers, add them to a list of the allowed additional headers. Restrictions: Spaces, wildcard characters (*), ampersands (&), colons (:), and less-than signs (<) are not allowed. Default value: None |
MaxAgeSeconds |
int |
Explanation: MaxAgeSeconds in the CORS rules of the bucket. It specifies the time your client can cache the response for a cross-origin request. Restrictions: Each CORS rule can specify only one value for MaxAgeSeconds. Value range: 0 to (231 – 1), in seconds Default value: 100 |
ContentLength |
int64 |
Explanation: Object size in bytes Value range: The value ranges from 0 TB to 48.8 TB, in bytes. Default value: None |
ContentType |
string |
Explanation: MIME type of the file to be downloaded. MIME type is a standard way of describing a data type and is used by the browser to decide how to display data. Value range: See What Is Content-Type (MIME)? Default value: If you do not specify this parameter when uploading an object, the SDK determines the object type based on the suffix of the specified object name and automatically assigns a value to this parameter. |
LastModified |
time.Time |
Explanation: Time when the last modification was made to the object Restrictions: The time must be in the ISO8601 format, for example, 2018-01-01T00:00:00.000Z. Default value: None |
ETag |
string |
Explanation: Base64-encoded, 128-bit MD5 value of an object. ETag is the unique identifier of the object content. It can be used to determine whether the object content is changed. For example, if the ETag value is A when an object is uploaded, but changes to B when the object is downloaded, it indicates that the object content has been changed. The ETag reflects changes to the object content, rather than the object metadata. An uploaded or copied object has a unique ETag after being encrypted using MD5. Restrictions: If an object is encrypted using server-side encryption, the ETag is not the MD5 value of the object. Value range: The value must contain 32 characters. Default value: None |
VersionId |
string |
Explanation: Object version ID Value range: The value must contain 32 characters. Default value: None |
Restore |
string |
Explanation: Restore status of an object. For an Archive object that is being restored or has been restored, this header is returned. For example, ongoing-request="true" indicates that the object is being restored. ongoing-request="false", expiry-date="Wed, 7 Nov 2012 00:00:00 GMT" indicates that the object has been restored. expiry-date indicates when the restored object expires. Restrictions: This parameter is only available for Archive objects. Default value: None |
Expiration |
string |
Explanation: Expiration details of the object. Example: "expiry-date=\"Mon, 11 Sep 2023 00:00:00 GMT\"" Default value: None |
SseHeader |
Explanation: Server-side encryption header information. If SSE-C is used, see Table 7. If SSE-KMS is used, see Table 8. |
|
WebsiteRedirectLocation |
string |
Explanation: If the bucket is configured with website hosting, the request for obtaining the object can be redirected to another object in the bucket or an external URL. This parameter specifies the address the request for the object is redirected to. The request is redirected to object anotherPage.html in the same bucket: WebsiteRedirectLocation:/anotherPage.html The request is redirected to an external URL http://www.example.com/: WebsiteRedirectLocation:http://www.example.com/ OBS obtains the specified value from the header and stores it in the object metadata WebsiteRedirectLocation. Restrictions:
Default value: None |
Metadata |
map[string]string |
Explanation: Custom object metadata. You can add a header starting with x-obs-meta- in the request to define metadata. The custom metadata will be returned in the response when you retrieve the object or query the object metadata. Restrictions:
Default value: None |
Constant |
Default Value |
Description |
---|---|---|
StorageClassStandard |
STANDARD |
OBS Standard Features low access latency and high throughput and is used for storing massive, frequently accessed (multiple times a month) or small objects (< 1 MB) requiring quick response. |
StorageClassWarm |
WARM |
OBS Infrequent Access Used for storing data that is semi-frequently accessed (fewer than 12 times a year) but is instantly available when needed. |
StorageClassCold |
COLD |
OBS Archive Used for storing rarely accessed (once a year) data. |
Parameter |
Type |
Mandatory (Yes/No) |
Description |
---|---|---|---|
Encryption |
string |
Yes if used as a request parameter |
Explanation: SSE-C used for encrypting objects Value range: AES256, indicating objects are encrypted using SSE-C Default value: None |
Key |
string |
Yes if used as a request parameter |
Explanation: Key for encrypting the object when SSE-C is used Restrictions: The value is a Base64-encoded 256-bit key, for example, K7QkYpBkM5+hca27fsNkUnNVaobncnLht/rCB2o/9Cw=. Default value: None |
KeyMD5 |
string |
No if used as a request parameter |
Explanation: MD5 value of the key for encrypting objects when SSE-C is used. This value is used to check whether any error occurs during the transmission of the key. Restrictions: The value is encrypted by MD5 and then encoded by Base64, for example, 4XvB3tbNTN+tIEVa0/fGaQ==. Default value: None |
Parameter |
Type |
Mandatory (Yes/No) |
Description |
---|---|---|---|
Encryption |
string |
Yes if used as a request parameter |
Explanation: SSE-KMS used for encrypting objects Value range: kms, indicating objects are encrypted using SSE-KMS Default value: None |
Key |
string |
No if used as a request parameter |
Explanation: ID of the KMS master key when SSE-KMS is used Value range: Valid value formats are as follows:
In the preceding formats:
Default value:
|
Code Examples
This example downloads example/objectname from examplebucket in a resumable download.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
package main import ( "fmt" "os" obs "github.com/huaweicloud/huaweicloud-sdk-go-obs/obs" ) func main() { //Obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways. Using hard coding may result in leakage. //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html. ak := os.Getenv("AccessKeyID") sk := os.Getenv("SecretAccessKey") // (Optional) If you use a temporary AK/SK pair and a security token to access OBS, you are advised not to use hard coding to reduce leakage risks. You can obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways. // securityToken := os.Getenv("SecurityToken") // Enter the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use. endPoint := "https://obs.ap-southeast-1.myhuaweicloud.com" // Create an obsClient instance. // If you use a temporary AK/SK pair and a security token to access OBS, use the obs.WithSecurityToken method to specify a security token when creating an instance. obsClient, err := obs.New(ak, sk, endPoint/*, obs.WithSecurityToken(securityToken)*/) if err != nil { fmt.Printf("Create obsClient error, errMsg: %s", err.Error()) } input := &obs.DownloadFileInput{} // Specify a bucket name. input.Bucket = "examplebucket" // Specify the object (example/objectname as an example) to download. input.Key = "example/objectname" // Specify a local full path (/tmp/objectname as an example) for the specified object. If the path is left blank, the current working directory will be used by default. input.DownloadFile = "/tmp/objectname" // Specify whether to enable resumable download. Value true is used as an example. The default value is false, indicating that resumable download is disabled. input.EnableCheckpoint = true // Specify a part size, in bytes. This example sets each part to 9 MB. input.PartSize = 9 * 1024 * 1024 // Specify the maximum number of parts that can be concurrently downloaded. 5 is used as an example. input.TaskNum = 5 // Download the object using resumable download. output, err := obsClient.DownloadFile(input) if err == nil { fmt.Printf("Download file(%s) under the bucket(%s) successful!\n", input.Key, input.Bucket) fmt.Printf("StorageClass:%s, ETag:%s, ContentType:%s, ContentLength:%d, LastModified:%s\n", output.StorageClass, output.ETag, output.ContentType, output.ContentLength, output.LastModified) return } fmt.Printf("Download file(%s) under the bucket(%s) fail!\n", input.Key, input.Bucket) if obsError, ok := err.(obs.ObsError); ok { fmt.Println("An ObsError was found, which means your request sent to OBS was rejected with an error response.") fmt.Println(obsError.Error()) } else { fmt.Println("An Exception was found, which means the client encountered an internal problem when attempting to communicate with OBS, for example, the client was unable to access the network.") fmt.Println(err) } } |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot