Performing a Resumable Download

Updated on 2024-12-03 GMT+08:00

View PDF

NOTICE:

If you have any questions during development, post them on the Issues page of GitHub.

Downloading large files often fails due to poor network conditions or program breakdowns. It is a waste of resources to restart the download process upon a download failure, and the restarted download process may still suffer from the unstable network. To resolve such issues, you can use the API for resumable download, whose working principle is to divide the to-be-downloaded file into multiple parts and download them separately. The download result of each part is recorded in a checkpoint file in real time. Only when all parts are successfully downloaded, the result indicating a successful download is returned. Otherwise, an exception is thrown to remind you of calling the API again for re-downloading. Based on the download status of each part recorded in the checkpoint file, the re-downloading will download the parts failed to be downloaded previously, instead of downloading all parts. By virtue of this, resources are saved and efficiency is improved.

You can use download_file to perform a resumable download. The following table describes the parameters involved in this API.

Field	Type	Mandatory or Optional	Description
option	The context of the bucket. For details, see Configuring option.	Mandatory	Bucket parameter
key	char *	Mandatory	Object name
version_id	char *	Mandatory	Object version ID
download_file_config	obs_download_file_configuration *	Mandatory	For details about how to download files, see the obs_download_file_configuration member description in the following table.
get_conditions	obs_get_conditions *	Mandatory	For details about how to set the object filter conditions and read range, see Performing a Conditioned Download.
encryption_params	server_side_encryption_params *	Optional	Encryption setting of the uploaded object
handler	obs_download_file_response_handler *	Mandatory	Callback function
callback_data	void *	Optional	Callback data

The following table describes the structure of obs_download_file_configuration.

Field	Type	Mandatory or Optional	Description
downLoad_file	char *	Mandatory	Local path to which the object is downloaded. If the value is null, the downloaded object is saved in the directory where the program is executed.
part_size	uint64_t	Mandatory	Part size, in bytes. The value ranges from 5 MB (default) to 5 GB.
task_num	int	Mandatory	Maximum number of parts that can be concurrently downloaded. The default value is 1.
enable_check_point	int	Mandatory	Whether to enable the resumable upload mode. The default value is 0, which indicates that this mode is disabled.
check_point_file	char *	Mandatory	File used to record the download progress. This parameter is effective only in the resumable download mode. If the value is null, the file is in the same local directory as the downloaded object.

Sample code:

void downloadFileResultCallback(obs_status status,
                                char *resultMsg,
                                int partCountReturn,
                                obs_download_file_part_info * downloadInfoList,
                                void *callbackData);
//Declare the callback function.
static void test_download_file(char *filename, char *key)
{
    obs_status ret_status = OBS_STATUS_BUTT;
    uint64_t uploadSliceSize =5L * 1024 * 1024;                     
    // Create and initialize option.
    obs_options option;
    init_obs_options(&option);
    option.bucket_options.host_name = "<your-endpoint>";
    option.bucket_options.bucket_name = "<Your bucketname>";

    // Hard-coded or plaintext AK/SK are risky. For security purposes, encrypt your AK/SK and store them in the configuration file or environment variables. In this example, the AK/SK are stored in environment variables for identity authentication. Before running this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    option.bucket_options.access_key = getenv("ACCESS_KEY_ID");
    option.bucket_options.secret_access_key = getenv("SECRET_ACCESS_KEY");
    // Initialize getConditions.
    obs_get_conditions getConditions;
    memset_s(&getConditions,sizeof(obs_get_conditions),0,sizeof(obs_get_conditions));
    init_get_properties(&getConditions);
    // Resumable download object information
    obs_download_file_configuration downloadFileConfig;
    memset_s(&downloadFileConfig,sizeof(obs_download_file_configuration),0,
                    sizeof(obs_download_file_configuration));
    downloadFileConfig.check_point_file = NULL;
    downloadFileConfig.enable_check_point = 1;
    downloadFileConfig.part_size = uploadSliceSize;
    downloadFileConfig.task_num = 10;
    downloadFileConfig.downLoad_file= filename;
    // Set response callback function.
    obs_download_file_response_handler Handler =
    { 
        {&response_properties_callback, &response_complete_callback_for_multi_task},
        &downloadFileResultCallback
    };
    initialize_break_point_lock();
    download_file(&option, key, 0,&getConditions,0,&downloadFileConfig, 
            &Handler, &ret_status);
    deinitialize_break_point_lock();
    if (OBS_STATUS_OK == ret_status) {
        printf("test download file successfully. \n");
    }
    else
    {
        printf("test download file faied(%s).\n", obs_get_status_name(ret_status));
    }
}
//downloadFileResultCallback is used as an example here. The printf statements can be replaced with custom logging print statements.
void downloadFileResultCallback(obs_status status,
                                char *resultMsg,
                                int partCountReturn,
                                obs_download_file_part_info * downloadInfoList,
                                void *callbackData)
{
    int i=0;
    obs_download_file_part_info * pstDownloadInfoList = downloadInfoList;    
	printf("status return is %d(%s)\n", status, obs_get_status_name(status));
    printf("%s",resultMsg);
    printf("partCount[%d]\n",partCountReturn);
    for(i=0;i<partCountReturn;i++)
    {
        printf("partNum[%d],startByte[%llu],partSize[%llu],status[%d]\n",
        pstDownloadInfoList->part_num,
        pstDownloadInfoList->start_byte,
        pstDownloadInfoList->part_size,
        pstDownloadInfoList->status_return);
        pstDownloadInfoList++;
    }	
	if (callbackData) {
		obs_status* retStatus = (obs_status*)callbackData;
		(*retStatus) = status;
	}
}

NOTE:

The API for resumable download, which is implemented based on Performing a Multipart Upload, is an encapsulated and enhanced version of partial download.
This API saves resources and improves efficiency upon the re-download, and speeds up the download process by concurrently downloading parts. Because this API is transparent to users, users are free from concerns about internal service details, such as the creation and deletion of checkpoint files, division of objects, and concurrent download of parts.
The default value of the enable_check_point parameter is 0, which indicates that the resumable download mode is disabled. In such cases, the API for resumable download degrades to the simple encapsulation of partial download, and no checkpoint file will be generated.
check_point_file is valid only when enable_check_point is set to 1.
Use the task_num parameter to configure how many concurrent tasks there can be in downloading a single object.