Help Center/ Object Storage Service/ SDK Reference/ Python/ APIs Related to Multipart Upload (SDK for Python)/ Multipart Upload Overview (SDK for Python)
Updated on 2024-11-26 GMT+08:00
View PDF

Multipart Upload Overview (SDK for Python)

You can upload large files using multipart upload. Multipart upload is applicable to many scenarios, including:

  • Files to be uploaded are larger than 100 MB.
  • The network condition is poor. Connection to the OBS server is constantly down.
  • Sizes of files to be uploaded are uncertain.

A multipart upload consists of the following steps:

  1. Initiate a multipart upload (ObsClient.initiateMultipartUpload).
  2. Upload parts one by one or concurrently (ObsClient.uploadPart).
  3. Assemble parts (ObsClient.completeMultipartUpload) or abort the multipart upload (ObsClient.abortMultipartUpload).

This example shows a complete multipart upload, including initiating a multipart upload, uploading parts, and assembling parts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# -*- coding:utf-8 -*-
from obs import ObsClient,CompleteMultipartUploadRequest, CompletePart
import os
import traceback
# Obtain an AK and SK pair using environment variables or import the AK and SK pair in other ways. Using hard coding may result in leakage.
# Obtain an AK and SK pair on the management console. For details, see https://support.huaweicloud.com/eu/usermanual-ca/ca_01_0003.html.
ak = os.getenv("AccessKeyID")
sk = os.getenv("SecretAccessKey")
# (Optional) If you use a temporary AK and SK pair and a security token to access OBS, obtain them from environment variables.
# security_token = os.getenv("SecurityToken")
# Set server to the endpoint corresponding to the bucket. EU-Dublin is used here as an example. Replace it with the one in use.
server = "https://obs.eu-west-101.myhuaweicloud.eu" 

# Create an obsClient instance.
# If you use a temporary AK and SK pair and a security token to access OBS, you must specify security_token when creating an instance.
obsClient = ObsClient(access_key_id=ak, secret_access_key=sk, server=server)
try:
    #Bucket name
    bucketName = "examplebucket"
    #Object name
    objectKey = "objectname"
    # Specify the MIME type for the object.
    contentType = 'text/plain'
    # Initiate a multipart upload.
    resp = obsClient.initiateMultipartUpload(bucketName, objectKey,
                                             contentType=contentType)
    #Obtain the uploadId.
    uploadId = resp.body["uploadId"]
    #Specify the size of the part to upload.
    partSize = 512 * 1024 * 1024
    #Specify the part number.
    partNum = 1
    # Specify whether object indicates the file path. True is used here. The default value is False.
    isFile = True
    #Specify the local object file to upload.
    filepath = r"D:\tmp\file.txt"
    contentLength = os.path.getsize(filepath)
    #Start offset of a part in the source file.
    offset = 0
    etags = {}

    while offset < contentLength:
        partSize = min(partSize, (contentLength - offset));
        # Upload parts.
        resp1 = obsClient.uploadPart(bucketName, objectKey, partNum, uploadId, filepath, isFile, partSize, offset)
        etags[partNum] = resp1.body.etag
        offset = offset + partSize
        partNum = partNum + 1

    completes = []
    for i in range(1, partNum):
        completes.append(CompletePart(i, etags[i]))
    # Assemble parts.
    completeMultipartUploadRequest = CompleteMultipartUploadRequest(parts = completes)
    resp = obsClient.completeMultipartUpload(bucketName, objectKey, uploadId, completeMultipartUploadRequest)
    # If status code 2xx is returned, the API is called successfully. Otherwise, the API call fails.
    if resp.status < 300:
        print('Upload Part Succeeded')
        print('requestId:', resp.requestId)
        print('etag:', resp.body.etag)
    else:
        print('Upload Part Failed')
        print('requestId:', resp.requestId)
        print('errorCode:', resp.errorCode)
        print('errorMessage:', resp.errorMessage)
except:
    print('multPartsUpload Failed')
    print(traceback.format_exc())

Below lists other multipart upload operations: