On this page

Show all

Help Center/ Object Storage Service/ SDK Reference/ Android/ Object Upload/ Performing a Multipart Upload

Performing a Multipart Upload

Updated on 2024-12-03 GMT+08:00

View PDF

NOTICE:

If you have any questions during development, post them on the Issues page of GitHub. For details about parameters and usage of each API, see the API Reference.

To upload a large file, multipart upload is recommended. Multipart upload is applicable to many scenarios, including:

Files to be uploaded are larger than 100 MB.
The network condition is poor. Connection to the OBS server is constantly down.
Sizes of files to be uploaded are uncertain.

Multipart upload consists of three phases:

Initialize a multipart upload (ObsClient.initiateMultipartUpload).
Upload parts one by one or concurrently (ObsClient.uploadPart).
Combine parts (ObsClient.completeMultipartUpload) or abort the multipart upload (ObsClient.abortMultipartUpload).

Initiating a Multipart Upload

Before upload, you need to notify OBS of initializing a multipart upload. This operation will return an upload ID (globally unique identifier) created by the OBS server to identify the multipart upload. You can use this upload ID to initiate related operations, such as aborting a multipart upload, listing multipart uploads, and listing uploaded parts.

You can call ObsClient.initiateMultipartUpload to initialize a multipart upload.

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest("bucketname", "objectname");
ObjectMetadata metadata = new ObjectMetadata();
metadata.addUserMetadata("property", "property-value");
metadata.setContentType("text/plain");
request.setMetadata(metadata);
InitiateMultipartUploadResult result = obsClient.initiateMultipartUpload(request);

String uploadId = result.getUploadId();
Log.i("InitiateMultipartUpload", "\t" + uploadId);

NOTE:

Call InitiateMultipartUploadRequest to specify the name and owning bucket of the uploaded object.
In InitiateMultipartUploadRequest, you can specify the MIME type, storage class, and customized metadata for the object.
The upload ID of the multipart upload returned by InitiateMultipartUploadResult.getUploadId will be used in follow-up operations.
Due to HTTP coding restrictions, non-ASCII characters cannot be sent. If your request headers contain full-width characters, the SDK will URL encode these characters before sending the request. When you use a browser to access the object metadata, the browser automatically decodes the data.
If you do not need the SDK to decode for you, call InitiateMultipartUploadRequest.setIsEncodeHeaders(false) to disable auto encoding.

Uploading a Part

After initializing a multipart upload, you can specify the object name and upload ID to upload a part. Each upload part has a part number (ranging from 1 to 10000). For parts with the same upload ID, their part numbers are unique and identify their comparative locations in the object. If you use the same part number to upload two parts, the latter one being uploaded will overwrite the former. Except for the part last uploaded whose size ranges from 0 to 5 GB, sizes of the other parts range from 100 KB to 5 GB. Parts are uploaded in random order and can be uploaded through different processes or machines. OBS will combine them into the object based on their part numbers.

You can call ObsClient.uploadPart to upload a part.

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

List<PartEtag> partEtags = new ArrayList<PartEtag>();
// Upload the first part.
UploadPartRequest request = new UploadPartRequest("bucketname", "objectname");
// Set the upload ID.
request.setUploadId(uploadId);
// Set the part number, which ranges from 1 to 10000.
request.setPartNumber(1);
// Set the large file to be uploaded. localfile is the path of the local file to be uploaded. You need to specify the file name.
request.setFile(new File("localfile"));

// Set the part size.
request.setPartSize(5 * 1024 * 1024L);
UploadPartResult result = obsClient.uploadPart(request);
partEtags.add(new PartEtag(result.getEtag(), result.getPartNumber()));

// Upload the second part.
request = new UploadPartRequest("bucketname", "objectname");
// Set the upload ID.
request.setUploadId(uploadId);
// Set the part number.
request.setPartNumber(2);
// Set the large file to be uploaded.
request.setFile(new File("localfile"));
// Set the offset for the second part.
request.setOffset(5 * 1024 * 1024L);
// Set the part size.
request.setPartSize(5 * 1024 * 1024L);
result = obsClient.uploadPart(request);
partEtags.add(new PartEtag(result.getEtag(), result.getPartNumber()));

NOTE:

Except the part last uploaded, other parts must be larger than 100 KB. Part sizes will not be verified during upload because which one is last uploaded is not identified until parts are combined.
OBS will return ETags (MD5 values) of the received parts to users.
To ensure data integrity, set UploadPartRequest.setAttachMd5 to true to make the SDK automatically calculate the MD5 value (valid only when the data source is a local file) of each part and add the MD5 value to the Content-MD5 request header. The OBS server will compare the MD5 value contained by each part and that calculated by the SDK to verify the data integrity.
You can call UploadPartRequest.setContentMd5 to set the MD5 value of the uploaded data directly. If this value is set, the UploadPartRequest.setAttachMd5 parameter becomes ineffective.
Part numbers range from 1 to 10000. If a part number exceeds this range, OBS will return a 400 Bad Request error.

Combining Parts

After all parts are uploaded, call the API for combining parts to generate the object. Before this operation, valid part numbers and ETags of all parts must be sent to OBS. After receiving this information, OBS verifies the validity of each part one by one. After all parts pass the verification, OBS combines these parts to form the final object.

You can call ObsClient.completeMultipartUpload to combine parts.

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

List<PartEtag> partEtags = new ArrayList<PartEtag>();

// First part
PartEtag part1 = new PartEtag();
part1.setPartNumber(1);
part1.seteTag("etag1");
partEtags.add(part1);

// Second part
PartEtag part2 = new PartEtag();
part2.setPartNumber(2);
part2.seteTag("etag2");
partEtags.add(part2);

CompleteMultipartUploadRequest request = new CompleteMultipartUploadRequest("bucketname", "objectname", uploadId, partEtags);

obsClient.completeMultipartUpload(request);

NOTE:

In the preceding code, partEtags indicates the list of part numbers and ETags of uploaded parts. These parts are listed in ascending order by part number.
Part numbers can be inconsecutive.

Concurrently Uploading Parts

Multipart upload is mainly used for large file upload or when the network condition is poor. The following sample code shows how to concurrently upload parts involved in a multipart upload:

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
final String bucketName = "bucketname";
final String objectKey = "objectname";
// Create an instance of ObsClient.
final ObsClient obsClient = new ObsClient(ak, sk, endPoint);

// Initialize the thread pool.
ExecutorService executorService = Executors.newFixedThreadPool(20);
final File largeFile = new File("localfile");

// Initiate a multipart upload.
InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest(bucketName, objectKey);
InitiateMultipartUploadResult result = obsClient.initiateMultipartUpload(request);

final String uploadId = result.getUploadId();
Log.i("UploadPart", "\t"+ uploadId + "\n");

// Set the part size to 100 MB.
long partSize = 100 * 1024 * 1024L;
long fileSize = largeFile.length();

// Calculate the number of parts to be uploaded.
long partCount = fileSize % partSize == 0 ? fileSize / partSize : fileSize / partSize + 1;

final List<PartEtag> partEtags = Collections.synchronizedList(new ArrayList<PartEtag>());

// Concurrently upload parts.
for (int i = 0; i < partCount; i++)
{
    // Start position of a part in the file
    final long offset = i * partSize;
    // Part size
    final long currPartSize = (i + 1 == partCount) ? fileSize - offset : partSize;
    // Part number
    final int partNumber = i + 1;
    executorService.execute(new Runnable()
    {
        @Override
        public void run()
        {
            UploadPartRequest uploadPartRequest = new UploadPartRequest();
            uploadPartRequest.setBucketName(bucketName);
            uploadPartRequest.setObjectKey(objectKey);
            uploadPartRequest.setUploadId(uploadId);
            uploadPartRequest.setFile(largeFile);
            uploadPartRequest.setPartSize(currPartSize);
            uploadPartRequest.setOffset(offset);
            uploadPartRequest.setPartNumber(partNumber);
            
            UploadPartResult uploadPartResult;
            try
            {
                uploadPartResult = obsClient.uploadPart(uploadPartRequest);
                Log.i("UploadPart", "Part#" + partNumber + " done\n");
                partEtags.add(new PartEtag(uploadPartResult.getEtag(), uploadPartResult.getPartNumber()));
            }
            catch (ObsException e)
            {
                Log.e("UploadPart", e.getMessage(), e);
            }
        }
    });
}

// Wait until the upload is complete.
executorService.shutdown();
while (!executorService.isTerminated())
{
    try
    {
        executorService.awaitTermination(5, TimeUnit.SECONDS);
    }
    catch (InterruptedException e)
    {
        Log.e("UploadPart", e.getMessage(), e);
    }
}

// Combine parts.
CompleteMultipartUploadRequest completeMultipartUploadRequest =
    new CompleteMultipartUploadRequest(bucketName, objectKey, uploadId, partEtags);
obsClient.completeMultipartUpload(completeMultipartUploadRequest);

NOTE:

When uploading a large file, use UploadPartRequest.setOffset and UploadPartRequest.setPartSize to determine the start and end positions of each part.

Aborting a Multipart Upload

After a multipart upload is aborted, you cannot use its upload ID to perform any operation and the uploaded parts will be deleted by OBS.

When an object is being uploaded in multi-part mode or an object fails to be uploaded, parts are generated in the bucket. These parts occupy your storage space. You can cancel the multi-part uploading task to delete unnecessary parts, thereby saving the storage space.

You can call ObsClient.abortMultipartUpload to abort a multipart upload.

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

AbortMultipartUploadRequest request = new AbortMultipartUploadRequest("bucketname", "objectname", uploadId);

obsClient.abortMultipartUpload(request);

Listing Uploaded Parts

You can call ObsClient.listParts to list successfully uploaded parts of a multipart upload.

The following table describes the parameters involved in this API.

Parameter	Description	Method in OBS Android SDK
bucketName	Bucket name	ListPartsRequest.setBucketName
key	Object name	ListPartsRequest.setKey
uploadId	Upload ID, which globally identifies a multipart upload. The value is in the returned result of ObsClient.initiateMultipartUpload.	ListPartsRequest.setUploadId
maxParts	Maximum number of parts that can be listed per page.	ListPartsRequest.setMaxParts
partNumberMarker	Part number after which listing parts begins. Only parts whose part numbers are larger than this value will be listed.	ListPartsRequest.setPartNumberMarker

Listing parts in simple mode

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

//List the uploaded parts. uploadId is obtained from initiateMultipartUpload.        
ListPartsRequest request = new ListPartsRequest("bucketname", "objectname");
request.setUploadId(uploadId);
ListPartsResult result = obsClient.listParts(request);

for(Multipart part : result.getMultipartList()){
// Part number, specified when being uploaded
   Log.i("ListParts", "\t"+part.getPartNumber());
// Part size
   Log.i("ListParts","\t"+part.getSize());
// Part ETag
   Log.i("ListParts","\t"+part.getEtag());
// Time when the part was last uploaded
   Log.i("ListParts","\t"+part.getLastModified());
}

NOTE:

Information about a maximum of 1000 parts can be listed each time. If an upload of the specific upload ID contains more than 1000 parts and ListPartsResult.isTruncated is true in the returned result, not all parts are listed. In such cases, you can use ListPartsResult.getNextPartNumberMarker to obtain the start position for next listing.
If you want to obtain all parts involved in a specific upload ID, you can use the paging mode for listing.

Listing all parts

If the number of parts of a multipart upload is larger than 1000, you can use the following sample code to list all parts.

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

//List the uploaded parts. uploadId is obtained from initiateMultipartUpload.        
ListPartsRequest request = new ListPartsRequest("bucketname", "objectname");
request.setUploadId(uploadId);
ListPartsResult result;

do{
    result = obsClient.listParts(request);
    for(Multipart part : result.getMultipartList()){
// Part number, specified when being uploaded
       Log.i("ListParts","\t"+part.getPartNumber());
// Part size
       Log.i("ListParts","\t"+part.getSize());
// Part ETag
       Log.i("ListParts","\t"+part.getEtag());
// Time when the part was last uploaded
       Log.i("ListParts","\t"+part.getLastModified());
    }
    request.setPartNumberMarker(Integer.parseInt(result.getNextPartNumberMarker()));
}while(result.isTruncated());

Listing Multipart Uploads

You can call ObsClient.listMultipartUploads to list multipart uploads. The following table describes parameters involved in ObsClient.listMultipartUploads.

Parameter	Description	Method in OBS Android SDK
bucketName	Bucket name	ListMultipartUploadsRequest.setBucketName
prefix	Prefix that the object names in the multipart uploads to be listed must contain	ListMultipartUploadsRequest.setPrefix
delimiter	Character used to group object names involved in multipart uploads. If the object name contains the delimiter parameter, the character string from the first character to the first delimiter in the object name is grouped under a single result element, commonPrefix. (If a prefix is specified in the request, the prefix must be removed from the object name.)	ListMultipartUploadsRequest.setDelimiter
maxUploads	Maximum number of multipart uploads listed in the response body. The value ranges from 1 to 1000. If the value exceeds 1000, only 1,000 multipart uploads are returned.	ListMultipartUploadsRequest.setMaxUploads
keyMarker	Object name to start with when listing multipart uploads	ListMultipartUploadsRequest.setKeyMarker
uploadIdMarker	Upload ID after which the multipart upload listing begins. It is effective only when used with keyMarker so that multipart uploads after uploadIdMarker of keyMarker will be listed.	ListMultipartUploadsRequest.setUploadIdMarker

Listing multipart uploads in simple mode

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

ListMultipartUploadsRequest request = new ListMultipartUploadsRequest("bucketname");

MultipartUploadListing result = obsClient.listMultipartUploads(request);
for(MultipartUpload upload : result.getMultipartTaskList()){
    Log.i("ListMultipartUploads","\t" + upload.getUploadId());
    Log.i("ListMultipartUploads","\t" + upload.getObjectKey());
    Log.i("ListMultipartUploads","\t" + upload.getInitiatedDate());
}

NOTE:

Information about a maximum of 1000 multipart uploads can be listed each time. If a bucket contains more than 1000 multipart uploads and MultipartUploadListing.isTruncated is true, not all uploads are listed. In such cases, you can use MultipartUploadListing.getNextKeyMarker and MultipartUploadListing.getNextUploadIdMarker to obtain the start position for next listing.
If you want to obtain all multipart uploads in a bucket, you can list them in paging mode.

Listing all multipart uploads in paging mode

// Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
String endPoint = "https://your-endpoint";
String uploadId = "upload id from initiateMultipartUpload";
// Create an instance of ObsClient.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);

ListMultipartUploadsRequest request = new ListMultipartUploadsRequest("bucketname");
MultipartUploadListing result;

do{
    result = obsClient.listMultipartUploads(request);
    for(MultipartUpload upload : result.getMultipartTaskList()){
        Log.i("ListMultipartUploads","\t" + upload.getUploadId());
        Log.i("ListMultipartUploads","\t" + upload.getObjectKey());
        Log.i("ListMultipartUploads","\t" + upload.getInitiatedDate());
    }
    request.setKeyMarker(result.getNextKeyMarker());
    request.setUploadIdMarker(result.getNextUploadIdMarker());
}while(result.isTruncated());

Parent topic: Object Upload

Previous topic: Setting Object Properties

Next topic: Configuring Lifecycle Management

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot