Help Center/ Object Storage Service/ User Guide/ Object Management/ Object Management/ Copying Objects
Updated on 2024-10-24 GMT+08:00
View PDF
Share

Copying Objects

Scenarios

You can copy objects in OBS. An object of up to 5 GB can be copied by each operation. To copy an object larger than 5 GB, use the multipart upload API. Specifically, this copy operation allows you to:

  • Create a copy for an object.
  • Rename an object by creating a copy for it and deleting the source object.
  • Edit object metadata.

    Each object has metadata, which is a set of name-value pairs. There is system-defined metadata and user-defined metadata. Some system-defined metadata can also be controlled by users. When you upload an object, you can set its metadata.

    • After you upload the object, you can modify its metadata using an API. For details, see Modifying Object Metadata.
    • You can also create an object copy and set the metadata. In the copy operation, set the target object to be the same as the source object.

    Both user-controlled metadata and user-defined metadata are copied. OBS automatically resets the system-controlled metadata. In the copy request, you do not need to set such metadata. For example, when you copy an object, OBS resets the creation date of the copied object.

    If you want to modify metadata, even only one piece of metadata, that can be configured by users (defined by users or the system), specify all the metadata that can be configured by users on the source object in the request.

Prerequisites

You have the read permission for the source object and the read and write permissions for the destination bucket.

Important Notes

  • Objects cannot be copied across regions. For example, objects in a bucket in the CN North-Beijing1 region cannot be copied to a bucket in the CN North-Beijing4 region. For details about how to automatically copy objects across regions, see Cross-Region Replication.
  • An appendable object cannot be copied.
  • You can copy an object of up to 5 GB in a single operation. To copy an object larger than 5 GB, use the multipart upload API.
  • If you want to copy a large number of Deep Archive objects, you can set the storage class of the copied objects to Standard and then transition them to the Deep Archive storage class through lifecycle rules to lower the costs on PUT requests.
  • By default, an original object will be overwritten by its copy with the same name. After the copy is successful, only the new object can be downloaded. The original object will be deleted. To prevent data loss:
    • Before copying an object, ensure that there is no object with the same name as the object copy or the object with the same name is useless.
    • Enable versioning.

      If versioning is enabled, deleted or overwritten objects will be retained as historical versions. You can restore a historical version at any time.

  • OBS allows you to copy your service data stored in OBS to a specified region, but Huawei Cloud does not detect the stored data. You need to ensure the legal compliance of your use of OBS on your own. If cross-border transfer is required, ensure that your use complies with relevant laws and regulations.
  • You cannot determine whether a copy request is executed successfully only using status_code in the HTTP header. If 200 in status_code is returned, the server has received the request and starts to process the request. The body in the response shows whether the request is executed successfully. The request is executed successfully only when the body contains ETag; otherwise, the request fails to be executed.

Ways to Copy Objects

You can use APIs, SDKs, OBS Browser+, or obsutil to copy objects.

Using the API

Copying Objects

Using SDKs

Java

Python

C

Go

BrowserJS

.NET

Android

iOS

PHP

Node.js

Using the GUI Tool - OBS Browser+

  1. Log in to OBS Browser+.
  2. Go to the target bucket. Right-click the file or folder and choose Copy from the shortcut menu.
  3. Right-click the path for saving the file or folder and choose Paste from the shortcut menu.

    • Files or folders in external buckets cannot be copied.
    • The restore status of Archive objects cannot be copied.
    • The source path and target path must be different.

Using the CLI Tool - obsutil

Command Line Structure

  • In Windows
    • Copying a single object 
      obsutil cp obs://srcbucket/key obs://dstbucket/[dest] [-dryRun][-u] [-crr] [-vlength] [-vmd5] [-p=1] [-threshold=52428800] [-versionId=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Copying objects in batches 
      obsutil cp obs://srcbucket[/key] obs://dstbucket[/dest] -r [-dryRun][-f] [-flat] [-u] [-crr] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
  • In Linux or macOS
    • Copying a single object 
      ./obsutil cp obs://srcbucket/key obs://dstbucket/[dest] [-dryRun] [-u] [-crr] [-vlength] [-vmd5] [-p=1] [-threshold=52428800] [-versionId=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-cpd=xxx] [-fr] [-o=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
    • Copying objects in batches 
      ./obsutil cp obs://srcbucket[/key] obs://dstbucket[/dest] -r [-dryRun] [-f] [-flat] [-u] [-crr] [-vlength] [-vmd5] [-j=1] [-p=1] [-threshold=52428800] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-ps=auto] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-mf] [-o=xxx] [-cpd=xxx] [-config=xxx] [-e=xxx] [-i=xxx] [-k=xxx] [-t=xxx]
  • The source path and destination path cannot be the same.
  • The source path and destination path cannot be partly overlapped either. If the source path overlaps with the prefix of the destination path, recursive replication applies. If the destination path overlaps with the prefix of the source path, the replication may overwrite objects in the source path.

Examples

  • Take the Windows OS as an example. Run the obsutil cp obs://bucket-test/key obs://bucket-test2 command to copy a single object. 
    obsutil cp obs://bucket-test/key obs://bucket-test2

Parallel:      3                   Jobs:          3
Threshold:     524288000           PartSize:      5242880
Exclude:                           Include:
VerifyLength:  false               VerifyMd5:     false
CheckpointDir: xxxx

[=====================================================] 100.00% 6/s 0s
Copy successfully, 19B, obs://bucket-test/key --> obs://bucket-test2/key
ext.txt
  • Take the Windows OS as an example. Run the obsutil cp obs://bucket-test/temp/ obs://bucket-test2 -f -r command to copy objects in batches. 
    obsutil cp obs://bucket-test/temp/ obs://bucket-test2 -r -f

Parallel:      3                   Jobs:          3
Threshold:     524288000           PartSize:      5242880
Exclude:                           Include:
VerifyLength:  false               VerifyMd5:     false
CheckpointDir: xxxx
OutputDir: xxxx


[=============================================================] 100.00% 10/s 0s
Succeed count is:   5         Failed count is:    0
Metrics [max cost:298 ms, min cost:192 ms, average cost:238.00 ms, average tps:9.71]
Task id is: 0476929d-9d23-4dc5-b2f8-0a0493f027c5
  • For more examples, see Copy.

Parameter Description

Parameter

Optional or Mandatory

Description

srcbucket

Mandatory

Source bucket name

dstbucket

Mandatory

Destination bucket name

dest

Optional

Indicates the destination object name when copying an object, or the name prefix of destination objects when copying objects in batches.

key

Mandatory for copying an object.

Optional for copying objects in batches.

Indicates the source object name when copying an object, or the name prefix of source objects when copying objects in batches.

The rules are as follows:

  • This parameter cannot be left blank when copying an object. If dest is left blank, the source object is copied to the root directory of the destination bucket. If the value of dest ends with a slash (/), the destination object name is the value of dest plus the source object name. Otherwise, the destination object name is the value of dest.
  • If this parameter is left blank when copying objects in batches, all objects in the source bucket are copied. If not, objects whose name prefix is the set value in the source bucket are copied. The rules for confirming the name of the destination object are as follows:
    • If the value of dest ends with a slash (/), the destination object name is the value of dest plus the source object name.
    • If the value of dest does not end with a slash (/), the destination object name is the value of dest/source object name.
NOTE:
  • If this parameter is configured but the flat parameter is not when copying objects in batches, the name of the source object contains the name prefix of the parent object. If flat is configured, then the name of the source object does not contain the name prefix of the parent object.
  • For details about how to use this parameter, see Copy.

fr

Optional for copying an object (additional parameter)

Generates an operation result list when copying an object.

flat

Optional for copying objects in batches (additional parameter)

The name prefix of the parent object is excluded when copying objects in batches.

dryRun

Optional (additional parameter)

Conducts a dry run.

crr

Optional (additional parameter)

Enables the client-side cross-region replication function. In this mode, data is directly copied to the destination bucket from the source bucket through data stream. The buckets can by any two OBS buckets.

NOTE:
  • If this parameter is configured, the configuration of client-side cross-region replication must be updated in the configuration file. For details, see Updating a Configuration File.
  • The configurations of the source bucket and destination bucket are respectively akCrr/skCrr/tokenCrr/endpointCrr and ak/sk/token/endpoint in the configuration file.
NOTICE:

When cross-region replication is enabled, the upload/download bandwidth, CPU, and memory resources of the host where commands are executed will be occupied, which may deteriorate the host performance.

vlength

Optional (additional parameter)

Verifies whether the object size in the destination bucket is the same as that in the source bucket after the copy task completes.

NOTE:

This parameter must be used together with crr.

vmd5

Optional (additional parameter)

Verifies whether the MD5 value of the destination bucket is the same as that of the source bucket after the copy task completes.

NOTE:
  • This parameter must be used together with crr.
  • Objects in the source bucket must contain metadata x-obs-meta-md5chksum, or MD5 verification will be skipped.

    After the MD5 verification is successful, this parameter value is used for metadata x-obs-meta-md5chksum of the destination object, for later MD5 verification during download or copy.

u

Optional (additional parameter)

Indicates incremental copy. If this parameter is set, each object can be copied only when it does not exist in the destination bucket, its size is different from the namesake one in the destination bucket, or it has the latest modification time.

p

Optional (additional parameter)

Indicates the maximum number of concurrent multipart copy tasks when copying an object. The default value is the value of defaultParallels in the configuration file.

threshold

Optional (additional parameter)

Indicates the threshold for enabling multipart copy, in bytes. The default value is the value of defaultBigfileThreshold in the configuration file.

NOTE:
  • If the size of the object to be copied is smaller than the threshold, copy the object directly. If not, a multipart copy is required.
  • If you copy an object directly, no part record is generated, and resumable transmission is not supported.
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).

versionId

Optional for copying an object (additional parameter)

Source object version ID that can be specified when copying an object

acl

Optional (additional parameter)

Access control policies for destination objects that can be specified when copying objects. Possible values are:

  • private
  • public-read
  • public-read-write
  • bucket-owner-full-control
NOTE:

The preceding four values indicate private read and write, public read, public read and write, and bucket owner full control.

sc

Optional (additional parameter)

Storage classes of the destination objects that can be specified when copying objects. Possible values are:

  • standard: Standard storage class. It features low access latency and high throughput, and is applicable to storing frequently accessed data (multiple accesses per month) or data that is smaller than 1 MB.
  • warm: Infrequent Access storage class. It is ideal for storing infrequently accessed (less than 12 times a year) data, but when needed, the access has to be fast.
  • cold: Archive storage class. It provides secure, durable, and inexpensive storage for rarely-accessed (once a year) data.
  • deep-archive: Deep Archive storage class (under limited beta testing). It is suitable for storing data that is barely (once every few years) accessed. This storage class costs less than the Archive storage class, but takes longer time (usually several hours) to restore data.

meta

Optional (additional parameter)

Standard or custom metadata that can be specified for destination objects in object replication. This parameter should be configured in the following format: key1:value1#key2:value2#key3:value3.

NOTE:
  1. The format example above indicates that the destination objects contain three groups of custom metadata: key1:value1, key2:value2, and key3:value3.
  2. Standard metadata headers include Content-Type, Content-Encoding, Cache-Control, Content-Disposition, Content-Language and Expires.

fs

Optional (additional parameter)

Specifies whether the method of listing parallel file systems is applied. If you are listing parallel file systems, you are recommended to add this parameter.

CAUTION:
  • With this method, the listing time required varies largely depending on the directory structures.
  • After this parameter is enabled, marker and limit will be ignored. Then, the buckets or parallel file systems (including directories) will be calculated.
  • This parameter is only supported by obsutil 5.5.12 and later.

ps

Optional (additional parameter)

Indicates the size of each part in a multipart copy task, in bytes. The value ranges from 100KB to 5GB. The default value is the value of defaultPartSize in the configuration file.

NOTE:
  • This parameter value can contain a unit, for example, 1MB (indicating 1048576 bytes).
  • The parameter can be set to auto. In this case, obsutil automatically sets the part size for each multipart task based on the source object size.

cpd

Optional (additional parameter)

Indicates the folder where the part records reside. The default value is .obsutil_checkpoint, the subfolder in the home directory of the user who executes obsutil commands.

NOTE:

A part record is generated during a multipart copy and saved to the copy subfolder. After the copy succeeds, its part record is deleted automatically. If the copy fails or is suspended, the system attempts to resume the task according to its part record when you perform the copy the next time.

r

Mandatory for copying objects in batches (additional parameter)

Copies objects in batches based on a specified name prefix of objects in the source bucket.

f

Optional for copying objects in batches (additional parameter)

Runs in force mode.

j

Optional for copying objects in batches (additional parameter)

Indicates the maximum number of concurrent tasks for copying objects in batches. The default value is the value of defaultJobs in the configuration file.

NOTE:

The value is ensured to be greater than or equal to 1.

exclude

Optional for copying objects in batches (additional parameter)

Indicates the matching patterns of source objects that are excluded, for example: *.txt.

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character. For instance, abc*.txt indicates any file whose name starts with abc and ends with .txt.
  • You can use \* to represent * and \? to represent ?.
  • If the name of the object to be copied matches the value of this parameter, the object is skipped.
NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and quotation marks for Windows.
  • The matching pattern applies to the absolute path of an object, including the object name prefix and object name starting from the root directory. For example, if the path of an object in the bucket is obs://bucket/src1/src2/test.txt, then the absolute path of the object is src1/src2/test.txt.
  • This matching pattern applies only to objects whose names do not end with a slash (/).
  • Multiple exclude parameters can be specified, for example, -exclude=*.xxx -exclude=*.xxx.

include

Optional for copying objects in batches (additional parameter)

Indicates the matching patterns of source objects that are included, for example: *.jpg.

NOTE:
  • The asterisk (*) represents any group of characters, and the question mark (?) represents any single character.
  • You can use \* to represent * and \? to represent ?.
  • Only after identifying that the name of the file to be copied does not match the value of exclude, the system checks whether the file name matches the value of this parameter. If yes, the file is copied. If not, the file is skipped.
NOTICE:
  • You are advised to use quotation marks for the matching pattern to prevent special characters from being escaped by the OS and leading to unexpected results. Use single quotation marks for Linux or macOS and quotation marks for Windows.
  • The matching pattern applies to the absolute path of an object, including the object name prefix and object name starting from the root directory. For example, if the path of an object in the bucket is obs://bucket/src1/src2/test.txt, then the absolute path of the object is src1/src2/test.txt.
  • This matching pattern applies only to objects whose names do not end with a slash (/).
  • Multiple include parameters can be specified, for example, -include=*.xxx -include=*.xxx.

timeRange

Optional for copying objects in batches (additional parameter)

Indicates the time range matching pattern when copying objects. Only objects whose latest modification time is within the configured time range are copied.

This pattern has a lower priority than the object matching patterns (exclude/include). That is, the time range matching pattern is executed after the configured object matching patterns.

NOTE:
  • The matching time range is represented in time1-time2, where time1 must be earlier than or the same as time2. The time format is yyyyMMddHHmmss.
  • Automatic formatting is supported. For example, yyyyMMdd is equivalent to yyyyMMdd000000, and yyyyMM is equivalent to yyyyMM01000000.
  • If this parameter is set to *-time2, all files whose latest modification time is earlier than time2 are matched. If it is set to time1-*, all files whose latest modification time is later than time1 are matched.
NOTICE:
  • Time in the matching pattern is the UTC time.
  • This matching pattern applies only to objects whose names do not end with a slash (/).

mf

Optional (additional parameter)

Indicates that the name matching pattern (include or exclude) and the time matching pattern (timeRange) also take effect on objects whose names end with a slash (/).

o

Optional (additional parameter)

Indicates the folder where operation result lists reside. After the command is executed, result lists (possibly including success, failure, and warning files) are generated in the folder. The default value is .obsutil_output, the subfolder in the home directory of the user who executes obsutil commands.

NOTE:
  • The naming rule for result lists is as follows: cp_{succeed | failed | warning}_report_time_TaskId.txt
  • By default, the maximum size of a single result list is 30 MB and the maximum number of result lists that can be retained is 1024. You can set the maximum size and number by configuring recordMaxLogSize and recordBackups in the configuration file.
  • If there are multiple folders and files and you need to confirm the detailed error information about a failed task, refer to the failure list cp_failed_report_time_TaskId.txt in the result list folder and the log files in the log path.

config

Optional (additional parameter)

User-defined configuration file for executing the current command. To learn the parameters that can be configured in this file, see Configuration Parameters.

e

Optional (additional parameter)

Specifies the endpoint.

i

Optional (additional parameter)

Specifies the user's AK.

k

Optional (additional parameter)

Specifies the user's SK.

t

Optional (additional parameter)

Specifies the user's security token.

Response

Field

Description

Parallel

Parameter -p in the request

Jobs

Parameter -j in the request

Threshold

Parameter -threshold in the request

PartSize

Parameter -ps in the request

Exclude

Parameter -exclude in the request

Include

Parameter -include in the request

TimeRange

Parameter -timeRange in the request

VerifyLength

Parameter -vlength in the request

VerifyMd5

Parameter -vmd5 in the request

CheckpointDir

Parameter -cpd in the request

OutputDir

Parameter -o in the request

ArcDir

Parameter -arcDir in the request

Succeed count

Number of successful tasks

Failed count

Number of failed tasks

Skip count

Number of tasks that are skipped during incremental upload, download, or copy, and synchronous upload, download, or copy.

NOTE:

Skipped tasks are recorded into successful tasks.

Warning count

Number of tasks that are executed successfully but contain warnings.

NOTE:
  • The task for which a warning is generated may be a failure or a success, which needs to be further determined according to the corresponding result list.
  • The number of tasks that generate warnings is independent of the number of successful or failed tasks. The total number of tasks is the number of successful tasks plus the number of failed tasks.

Succeed bytes

Number of bytes that are successfully uploaded or downloaded.

max cost

Maximum duration of all tasks, in ms

min cost

Minimum duration of all tasks, in ms

average cost

Average duration of all tasks, in ms

average tps

The average number of tasks completed per second

Task id

Unique ID of an operation, which is used to search for the result list generated in a batch task
Parent Topic: Object Management