Copying Objects in Batches

Function

The object copy operation can create a copy for an existing object in OBS.

You can use the APIs for listing and copying objects to copy objects in batches. When copying objects, you can specify the properties of the new objects and set object permissions. In addition, copying objects by specifying conditions is supported.

Sample Code

This example copies objects in batches.

// Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK before storing them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables. Before running the code in this example, configure environment variables AccessKeyID and SecretAccessKey.
// The front-end code does not have the process environment variable, so you need to use a module bundler like webpack to define the process variable.
// Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
const AK = process.env.AccessKeyID
const SK = process.env.SecretAccessKey
// (Optional) If you use a temporary AK and SK pair and a security token to access OBS, obtain them using environment variables.
const security_token= process.env.SecurityToken
// Set server to the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use.
const server = "https://obs.ap-southeast-1.myhuaweicloud.com"

// Create an ObsClient instance.
var obsClient = new ObsClient({
    access_key_id: AK,
    secret_access_key: SK,
    server: server
});

// Source bucket name
var sourceBucket = 'examplebucket';
// Destination bucket name
var destBucket = 'destexamplebucket';
// Prefix of the objects to be copied. If the prefix is left blank, all objects will be copied.
var prefix = '';
// Prefix of the destination objects. The prefix is added after the objects are copied.
var destPrefix = 'backup/';
async function copyObjects() {
    var marker = '';
    var totalCount = 0;
    console.log('Starting listing and copying objects...');
    console.log('Source bucket: ' + sourceBucket + ', destination bucket: ' + destBucket);
    while (true) {
        var listParams = { Bucket: sourceBucket };
        if (prefix) {
            listParams.Prefix = prefix;
        }
        if (marker) {
            listParams.Marker = marker;
        }
        var listResult = await obsClient.listObjects(listParams);
        if (listResult.CommonMsg.Status <= 300) {
            if (listResult.InterfaceResult.Contents) {
                var batchCount = listResult.InterfaceResult.Contents.length;
                console.log ('Obtained ' + batchCount + ' objects in this batch. Starting copy...');
                for (var i = 0; i < listResult.InterfaceResult.Contents.length; i++) {
                    var sourceKey = listResult.InterfaceResult.Contents[i].Key;
                    var destKey = destPrefix + sourceKey;
                    totalCount++;
                    try {
                        var params = {
                            Bucket: destBucket,
                            Key: destKey,
                            CopySource: sourceBucket + '/' + sourceKey
                        };
                        var result = await obsClient.copyObject(params);
                        if (result.CommonMsg.Status <= 300) {
                            console.log('[' + totalCount + '] succeeded: ' + sourceKey + ' -> ' + destKey);
                        } else {
                            console.log('[' + totalCount + '] failed: ' + sourceKey + ' - ' + result.CommonMsg.Message);
                        }
                    } catch (error) {
                        console.log('[' + totalCount + '] exception occurred: ' + sourceKey + ' - ' + error);
                    }
                }
            }
            if (listResult.InterfaceResult.IsTruncated == 'true') {
                marker = listResult.InterfaceResult.NextMarker;
            } else {
                break;
            }
        } else {
            console.log('Failed to list objects: ', listResult.CommonMsg.Message);
            return;
        }
    }
    console.log('Batch copy completed. A total of ' + totalCount + ' objects are processed.');
}
copyObjects()

Use the CopySource parameter to specify the information about the source objects.