Updated on 2024-04-29 GMT+08:00

Performing a Resumable Upload

If you have any questions during development, post them on the Issues page of GitHub.

Uploading large files often fails due to poor network conditions or program breakdowns. It is a waste of resources to restart the upload process upon an upload failure, and the restarted upload process may still suffer from the unstable network. To resolve such issues, you can use the API for resumable upload, whose working principle is to divide the to-be-uploaded file into multiple parts and upload them separately. The upload result of each part is recorded in a checkpoint file in real time. Only when all parts are successfully uploaded, the result indicating a successful upload is returned. Otherwise, an exception is thrown to remind you of calling the API again for re-uploading. Based on the upload status of each part recorded in the checkpoint file, the re-uploading will upload the parts failed to be uploaded previously, instead of uploading all parts. By virtue of this, resources are saved and efficiency is improved.

You can call upload_file to perform a resumable upload. The following table describes the parameters involved in this API.

Description

Field

Type

Mandatory or Optional

Description

option

The context of the bucket. For details, see Configuring option.

Mandatory

Bucket parameter

key

char *

Mandatory

Object name

upload_file_config

obs_upload_file_configuration *

Mandatory

For details about the configuration of the file to be uploaded, see the following table.

encryption_params

server_side_encryption_params *

Optional

Encryption setting of the uploaded object

handler

obs_upload_file_response_handler *

Mandatory

Callback body. All member variables are pointers to the callback function.

callback_data

void *

Optional

Callback data

The following table describes the structure of obs_upload_file_configuration.

Member Name

Type

Mandatory or Optional

Description

upload_file

char *

Mandatory

Local file to be uploaded

part_size

uint64_t

Mandatory

Part size, in bytes. The value ranges from 100 KB to 5 GB. The default value is 5 MB.

check_point_file

char *

Mandatory

File used to record the upload progress. This parameter is effective only in the resumable upload mode. If the value is null, the file is in the same directory as the local file to be uploaded.

enable_check_point

int

Mandatory

Whether to enable the resumable upload mode. The default value is 0, which indicates that this mode is disabled.

task_num

int

Mandatory

Maximum number of parts that can be concurrently uploaded. The default value is 1.

Sample Code

Sample code:

void uploadFileResultCallback(obs_status status,
                              char *resultMsg,
                              int partCountReturn,
                              obs_upload_file_part_info * uploadInfoList,
                              void *callbackData);
//Declare the callback function.
static void test_upload_file()
{
    obs_status ret_status = OBS_STATUS_BUTT;
    // Create and initialize option.
    obs_options option;
    init_obs_options(&option);
    option.bucket_options.host_name = "<your-endpoint>";
    option.bucket_options.bucket_name = "<Your bucketname>";

    // Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    option.bucket_options.access_key = getenv("ACCESS_KEY_ID");
    option.bucket_options.secret_access_key = getenv("SECRET_ACCESS_KEY");
    // Initialize the put_properties structure.
    obs_put_properties put_properties;
    init_put_properties(&put_properties);

    obs_upload_file_configuration uploadFileInfo;
    memset_s(&uploadFileInfo,sizeof(obs_upload_file_configuration),0,sizeof(obs_upload_file_configuration));
    uploadFileInfo.check_point_file = 0;
    uploadFileInfo.enable_check_point = 1;
    uploadFileInfo.part_size = "<part size>";
    uploadFileInfo.task_num = "<task num>";
    uploadFileInfo.upload_file = "<upload filename>";
    uploadFileInfo.put_properties = &put_properties;  

    // Callback function
    obs_upload_file_response_handler Handler =
    { 
        {&response_properties_callback, &response_complete_callback_for_multi_task},
        &uploadFileResultCallback
    };    
    initialize_break_point_lock();
    upload_file(&option, "<Your Key>", 0, &uploadFileInfo, Null, &Handler, &ret_status);
    deinitialize_break_point_lock();
    if (OBS_STATUS_OK == ret_status) {
        printf("test upload file successfully. \n");
    }
    else
    {
        printf("test upload file faied(%s).\n", obs_get_status_name(ret_status));
    }
}
//uploadFileResultCallback is used as an example here. The printf statements can be replaced with custom logging print statements.
void uploadFileResultCallback(obs_status status,
                              char *resultMsg,
                              int partCountReturn,
                              obs_upload_file_part_info * uploadInfoList,
                              void *callbackData)
{
    int i=0;
    obs_upload_file_part_info * pstUploadInfoList = uploadInfoList;
    printf("status return is %d(%s)\n",status,obs_get_status_name(status));
    printf("%s",resultMsg);
    printf("partCount[%d]\n",partCountReturn);
    for(i=0;i<partCountReturn;i++)
    {
        printf("partNum[%d],startByte[%llu],partSize[%llu],status[%d]\n",
        pstUploadInfoList->part_num,
        pstUploadInfoList->start_byte,
        pstUploadInfoList->part_size,
        pstUploadInfoList->status_return);
        pstUploadInfoList++;
    }
	if (callbackData) {
		obs_status* retStatus = (obs_status*)callbackData;
		(*retStatus) = status;
	}
}
  • The API for resumable upload, which is implemented based on Performing a Multipart Upload, is an encapsulated and enhanced version of multipart upload.
  • This API saves resources and improves efficiency upon the re-upload, and speeds up the upload process by concurrently uploading parts. Because this API is transparent to users, users are free from concerns about internal service details, such as the creation and deletion of checkpoint files, division of objects, and concurrent upload of parts.
  • enable_check_point: The value 0 indicates that the resumable upload mode is disabled. In this case, the resumable upload API functions as an encapsulated version of multipart upload and does not generate checkpoint files. The value 1 indicates that the resumable upload mode is enabled.