Cross-Region Replication Across Accounts Using OMS

srcbucket

Mandatory

The source bucket name

dstbucket

Mandatory

The destination bucket name

dest

Optional

The destination object name when copying an object, or the name prefix of destination objects when copying objects in batches

key

Mandatory for copying an object.

Optional for copying objects in batches.

The source object name when copying an object, or the name prefix of source objects when copying objects in batches

The rules are as follows:

• This parameter cannot be left blank when copying an object.
  • If dest is left blank, the source object is copied to the root directory of the destination bucket.
  • If the value of dest ends with a slash (/), the destination object name is the value of dest plus the source object name. Otherwise, the destination object name is the value of dest.
• If this parameter is left blank when copying objects in batches, all objects in the source bucket are copied. If not, objects whose name prefix is the set value in the source bucket are copied. The rules for confirming the name of the destination object are as follows:
  • If the value of dest ends with a slash (/), the destination object name is the value of dest plus the source object name.
  • If the value of dest does not end with a slash (/), the destination object name is the value of dest/source object name.
• If this parameter is configured but the flat parameter is not when copying objects in batches, the name of the source object contains the name prefix of the parent object. If flat is configured, then the name of the source object does not contain the name prefix of the parent object.

fr

Optional for copying an object (additional parameter)

Generates an operation result file when copying an object.

flat

Optional for copying objects in batches (additional parameter)

The name prefix of the parent object is excluded when copying objects in batches.

dryRun

Optional (additional parameter)

Conducts a dry run.

crr

Mandatory for cross-region replication across accounts

Enables the client-side cross-region replication function. In this mode, data is directly copied to the destination bucket from the source bucket through data stream. The buckets can be any two OBS buckets.

• If this parameter is configured, the configuration of client-side cross-region replication must be updated in the configuration file. For details, see Updating a Configuration File.
• When cross-region replication is enabled, the upload/download bandwidth, CPU, and memory resources of the host where commands are executed will be occupied, which may deteriorate the host performance.

vlength

Optional (additional parameter)

Verifies whether the object size in the destination bucket is the same as that in the source bucket after the copy task completes.

This parameter must be used together with crr.

vmd5

Optional (additional parameter)

Verifies whether the MD5 value of the destination bucket is the same as that of the source bucket after the copy task completes.

• This parameter must be used together with crr.
• Objects in the source bucket must contain metadata x-obs-meta-md5chksum, or MD5 verification will be skipped. After the MD5 verification is successful, this parameter value is used for metadata x-obs-meta-md5chksum of the destination object, for later MD5 verification during download or replication.
• If your object needs encryption, do not use this parameter.

u

Optional (additional parameter)

Indicates incremental copy. If this parameter is set, each object can be copied only when it does not exist in the destination bucket, its size is different from the namesake one in the destination bucket, or it has the latest modification time.

p

Optional (additional parameter)

The maximum number of concurrent multipart copy tasks when copying an object. The default value is the value of defaultParallels in the configuration file.

threshold

Optional (additional parameter)

The threshold for enabling multipart copy. If the size of the object to be copied is smaller than the threshold, copy the object directly. If not, a multipart copy is required.

The default value is the value of defaultBigfileThreshold in the configuration file.

Unit: byte

This parameter value can contain a unit, for example, 1MB (indicating 1,048,576 bytes).

versionId

Optional for copying an object (additional parameter)

The source object version ID that can be specified when copying an object

acl

Optional (additional parameter)

The access control policies for destination objects that can be specified when copying objects.

Possible values are:

• private: Only users granted permissions by the object ACL can access the object.
• public-read: Anyone can read objects in the bucket.
• public-read-write: Anyone can read, write, or delete objects in the bucket.
• bucket-owner-full-control: Only the bucket owner has full control over objects in the bucket.

sc

Optional (additional parameter)

The storage classes of the destination objects that can be specified when copying objects.

Possible values are:

• standard: The Standard storage class. It features low access latency and high throughput, and is suitable for storing frequently accessed data (multiple times a month on average) or data that is smaller than 1 MB.
• warm: The Infrequent Access storage class. It is suitable for storing infrequently accessed (less than 12 times a year on average) data, but when needed, the access has to be fast. It has a slightly lower availability than Standard.
• cold: The Archive storage class. It provides secure, durable, and inexpensive storage for rarely-accessed (once a year on average) data.
• deep-archive: The Deep Archive storage class. It is suitable for storing data that is hardly ever accessed (once every few years on average). This storage class costs less than the Archive storage class, but takes longer (usually several hours) to restore data.

meta

Optional (additional parameter)

The standard or user-defined metadata that can be specified during object replication.

• Standard metadata includes Content-Type, Content-Encoding, Cache-Control, Content-Disposition, Content-Language and Expires.
• User-defined metadata must be in the format of key:value. Multiple user-defined metadata items are separated by number signs (#), for example, key1:value1#key2:value2#key3:value3, which indicates that the object copy in the destination bucket contains three groups of user-defined metadata: key1:value1, key2:value2, and key3:value3.

fs

Optional (additional parameter)

Specifies whether the method of listing parallel file systems is applied. If you are listing parallel file systems, you are recommended to add this parameter.

• After this parameter is enabled, marker and limit will be ignored. Then, the size of the bucket, parallel file system, or directory will be calculated.
• This parameter is only supported for obsutil 5.5.12 or later.
• With this method, the listing time required varies largely depending on the directory structures.

ps

Optional (additional parameter)

The size of each part in a multipart copy task

• The default value is the value of defaultPartSize in the configuration file. The parameter can be set to auto. In this case, obsutil automatically configures the part size for each multipart task based on the source file size.
• Range: 100KB to 5GB
• Unit: byte. This parameter value can contain a unit, for example, 1MB (indicating 1,048,576 bytes).

cpd

Optional (additional parameter)

The folder where the part records reside. The default value is .obsutil_checkpoint, the subfolder in the home directory of the user who executes obsutil commands.

A part record is generated during a multipart copy and saved to the upload subfolder. After the copy succeeds, its part record is deleted automatically. If the copy fails or is suspended, the system attempts to resume the task according to its part record when you perform the copy the next time.

r

Mandatory for copying objects in batches (additional parameter)

Copies objects in batches based on a specified name prefix of objects in the source bucket.

f

Optional for copying objects in batches (additional parameter)

Runs in force mode.

j

Optional for copying objects in batches (additional parameter)

The maximum number of concurrent tasks for copying objects in batches. The default value is the value of defaultJobs in the configuration file.

exclude

Optional for copying objects in batches (additional parameter)

The matching patterns of source objects that are excluded. After this parameter is specified, if the name of the object to be copied matches the value of this parameter, the object is skipped. For example, if this parameter is set to *.txt, all files whose names end with .txt are skipped and not copied.

• An asterisk (*) matches any number of characters. For instance, abc*.txt matches all files whose names start with abc, contain any number of characters in the middle, and end with .txt.
• A question mark (?) matches any single character. For instance, abc?.txt matches all files whose names start with abc, contain any single character in the middle, and end with .txt.
• You can use \* to represent * and \? to represent ?.
• Multiple exclude parameters can be specified, for example, -exclude=*.xxx -exclude=*.xxx.

Constraints:

This matching pattern applies only to objects whose names do not end with a slash (/).

include

Optional for copying objects in batches (additional parameter)

The included matching patterns of source objects. After this parameter is specified, if the name of the object to be copied matches the value of this parameter, the object is copied. For example, if this parameter is set to *.txt, all objects whose names end with .txt are copied.

• An asterisk (*) matches any number of characters. For instance, abc*.txt matches all files whose names start with abc, contain any number of characters in the middle, and end with .txt.
• A question mark (?) matches any single character. For instance, abc?.txt matches all files whose names start with abc, contain any single character in the middle, and end with .txt.
• You can use \* to represent * and \? to represent ?.
• Multiple include parameters can be specified, for example, -include=*.xxx -include=*.xxx.

Constraints:

• This matching pattern applies only to objects whose names do not end with a slash (/).
• If both the exclude and include parameters are configured, the exclude rules are applied first. If the name of an object to be copied does not match exclude, OBS checks whether the object name matches the value of include. If yes, the object is copied. If no, the object is skipped.

Example:

./obsutil cp obs://src-bucket/ obs://target-bucket/ -include=/src-object/* -f -r

This command copies files whose names start with src-object/ from the src-bucket bucket to the target-bucket bucket.

timeRange

Optional for copying objects in batches (additional parameter)

The time range matching pattern when copying objects. Only objects whose latest modification time is within the configured time range are copied.

This pattern has a lower priority than the file matching patterns (exclude/include). That is, the time range matching pattern is executed after the configured file matching patterns.

• Time in the matching pattern is the UTC time.
• The matching time range is represented in time1-time2, where time1 must be earlier than or the same as time2. The time format is yyyyMMddHHmmss.
• Automatic formatting is supported. For example, yyyyMMdd is equivalent to yyyyMMdd000000, and yyyyMM is equivalent to yyyyMM01000000.
• If this parameter is set to *-time2, all files whose last modification time is earlier than time2 are matched. If it is set to time1-*, all files whose last modification time is later than time1 are matched.

This matching pattern applies only to objects whose names do not end with a slash (/).

mf

Optional (additional parameter)

Indicates that the name matching pattern (include or exclude) and the time matching pattern (timeRange) also take effect on objects whose names end with a slash (/).

o

Optional (additional parameter)

The folder that stores the result files. After the command is executed, result files (possibly success, failure, and warning files) will be created in the specified folder.

The default value is .obsutil_output, a subfolder in the user's home directory where obsutil commands are executed.

• A result file should be named as follows: cp_{succeed | failed | warning}_report_time_TaskId.txt.
• By default, the maximum size of a single result file is 30 MB. You can set the maximum size by configuring recordMaxLogSize in the configuration file.
• By default, the maximum number of result files that can be retained is 1,024. You can set the maximum number by configuring recordBackups in the configuration file.
• If there are multiple folders and files and you need to confirm the details of a failed task, refer to the failure result file cp_failed_report_time_TaskId.txt in the result folder and the log files in the log path.

config

Optional (additional parameter)

The user-defined configuration file for executing the current command.

To learn the parameters that can be configured in this file, see Configuration Parameters.

e

Optional (additional parameter)

The endpoint

i

Optional (additional parameter)

The user's AK

k

Optional (additional parameter)

The user's SK

t

Optional (additional parameter)

The user's security token

Parallel

The parameter -p in the request

Jobs

The parameter -j in the request

Threshold

The parameter -threshold in the request

PartSize

The parameter -ps in the request

Exclude

The parameter -exclude in the request

Include

The parameter -include in the request

TimeRange

The parameter -timeRange in the request

VerifyLength

The parameter -vlength in the request

VerifyMd5

The parameter -vmd5 in the request

CheckpointDir

The parameter -cpd in the request

OutputDir

The parameter -o in the request

ArcDir

The parameter -arcDir in the request

Succeed count

The number of successful tasks

Failed count

The number of failed tasks

Skip count

The number of tasks that are skipped during incremental upload, download, or copy, and synchronous upload, download, or copy.

Warning count

The number of tasks that are executed successfully but contain warnings.

Succeed bytes

The number of bytes that are successfully uploaded or downloaded.

max cost

The maximum duration of all tasks, in ms

min cost

The minimum duration of all tasks, in ms

average cost

The average duration of all tasks, in ms

average tps

The average number of tasks completed per second

Task id

The unique ID of an operation, which is used to search for the result file generated for a batch task