Multipart Upload Overview (SDK for Python)
You can upload large files using multipart upload. Multipart upload is applicable to many scenarios, including:
- Files to be uploaded are larger than 100 MB.
- The network condition is poor. Connection to the OBS server is constantly down.
- Sizes of files to be uploaded are uncertain.
A multipart upload consists of the following steps:
- Initiate a multipart upload (ObsClient.initiateMultipartUpload).
- Upload parts one by one or concurrently (ObsClient.uploadPart).
- Assemble parts (ObsClient.completeMultipartUpload) or abort the multipart upload (ObsClient.abortMultipartUpload).
This example shows a complete multipart upload, including initiating a multipart upload, uploading parts, and assembling parts.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | # -*- coding:utf-8 -*- import os import traceback from obs import ObsClient, CompleteMultipartUploadRequest, CompletePart # Obtain an AK/SK pair using environment variables (recommended) or import it in other ways. Using hard coding may result in leakage. # Obtain an AK and SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html. ak = os.getenv("AccessKeyID") sk = os.getenv("SecretAccessKey") # (Optional) If you use a temporary AK/SK pair and a security token to access OBS, obtain them from environment variables. # security_token = os.getenv("SecurityToken") # Set server to the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use. server = "https://obs.ap-southeast-1.myhuaweicloud.com" # Create an obsClient instance. # If you use a temporary AK/SK pair and a security token to access OBS, you must specify security_token when creating an instance. obs_client = ObsClient(access_key_id=ak, secret_access_key=sk, server=server) try: # Bucket name bucket_name = "examplebucket" # Object name object_key = "objectname" # MIME type of the object content_type = 'text/plain' # Initialize a multipart upload task. resp = obs_client.initiateMultipartUpload(bucket_name, object_key, content_type = content_type) # Obtain the upload ID of the initialized upload task. upload_id = resp.body["uploadId"] # Size of the part to be uploaded part_size = 512 * 1024 * 1024 # Part number part_num = 1 # Specify whether object indicates the file path. True is used here. The default value is False. is_file = True # Local object file to be uploaded filepath = r"D:\tmp\file.txt" content_length = os.path.getsize(filepath) # Start offset of a part in the source file offset = 0 etags = {} while offset < content_length: part_size = min(part_size, (content_length - offset)); # Upload parts. resp1 = obs_client.uploadPart(bucket_name, object_key, part_num, upload_id, filepath, is_file, part_size, offset) etags[part_num] = resp1.body.etag offset = offset + part_size part_num = part_num + 1 completes = [] for i in range(1, part_num): completes.append(CompletePart(i, etags[i])) # Assemble parts. complete_multipart_upload_request = CompleteMultipartUploadRequest(parts=completes) resp = obs_client.completeMultipartUpload(bucket_name, object_key, upload_id, complete_multipart_upload_request) # If status code 2xx is returned, the API call succeeds. Otherwise, the API call fails. if resp.status < 300: print('Upload Part Succeeded') print('requestId:', resp.requestId) print('etag:', resp.body.etag) else: print('Upload Part Failed') print('requestId:', resp.requestId) print('errorCode:', resp.errorCode) print('errorMessage:', resp.errorMessage) except Exception: print('multPartsUpload Failed') print(traceback.format_exc()) |
Below lists other multipart upload operations:
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot