Bucket Inventory

Updated on 2024-01-02 GMT+08:00

View PDF

Scenarios

A bucket inventory can list objects in a bucket, save the related object information in CSV files, and deliver the CSV files to the bucket specified for storing bucket inventory files. In this manner, you can easily manage objects in a bucket. A source bucket can also be the destination bucket.

A bucket inventory file can contain the following object related information: versions, sizes, storage classes, tags, encryption statuses, and last modification time.
You can encrypt bucket inventory files in the SSE-KMS mode.
You can set the frequency (daily or weekly) for generating bucket inventory files.
You can also specify a bucket to store the generated bucket inventory files.

Constraints

Bucket versions

Inventories can be generated only for OBS 3.0 buckets, but they can be stored in either OBS 3.0 or OBS 2.0 buckets.

Number of bucket inventories

A bucket can have a maximum of 10 inventories.

Source and destination buckets

The source bucket (for which a bucket inventory rule is configured) and the destination bucket (where the generated inventory files are stored) must belong to the same account.
The source bucket and the destination bucket must be in the same region.
Default encryption cannot be enabled for the destination bucket configured for storing inventory files.

Functions

Inventory files must be in the CSV format.
OBS can generate inventory files for all objects in a bucket or a group of objects whose names begin with the same prefix.
If a bucket has multiple inventory rules, these rules must not overlap.
- If a bucket already has an inventory rule for the entire bucket, new inventory rules that filter objects by prefixes cannot be created. If you need an inventory rule that covers only a subset of objects in the bucket, first delete the inventory rule configured for the entire bucket.
- If an inventory rule that filters objects by a specified prefix already exists, you cannot create an inventory rule for the entire bucket. To create an inventory rule for the entire bucket, make sure that the bucket has no other inventory rules that filter objects by specified prefixes.
- If a bucket already has an inventory rule that filters objects by the object name prefix ab, the filter of a new inventory rule cannot start with a or abc. Or, you can delete the existing inventory rule and create a new one that filters objects according to your needs.
Only SSE-KMS can be used to encrypt bucket inventories.

Permissions

Inventory files are delivered to the destination bucket by an OBS system user. Therefore, you need to authorize the system user the permission to write the destination bucket.

Others

The bucket inventory function is offered for free, but inventory files are billed for the storage space they occupy.

How Is a Bucket Inventory Configured?

Before the configuration, you need to briefly understand what a source bucket or a destination bucket is.

Source bucket: A source bucket is the bucket for which an inventory is configured. The inventory lists objects stored in the source bucket.
Destination bucket: A destination bucket is where generated inventory files are stored. A source bucket can also be the destination bucket. You can specify a name prefix for an inventory. Then generated inventory files will be named with the prefix and saved in the directory with the prefix. If you do not specify any name prefix for the inventory, the generated inventory files are stored in the root directory of the bucket.
- Restrictions on the destination bucket
  - The destination bucket and source bucket must belong to the same tenant.
  - The destination bucket and source bucket must be in the same region.
  - A bucket policy must be configured to grant OBS the permission to write objects to the destination bucket. For details, see Add a bucket policy for the destination bucket.
- The destination bucket contains the following files:
  - A list of inventory files
  - The Manifest file, which contains the list of all inventory files under a certain inventory configuration. For details about the Manifest file, see Manifest File.

Configuring a Bucket Inventory

You use OBS Console or call the API to configure a bucket inventory. If you configure a bucket inventory on OBS Console, a bucket policy with the required permission configuration is automatically generated for the destination bucket. If you call the API to configure the bucket inventory, you need to manually configure the bucket policy for the destination bucket.

Add a bucket policy for the destination bucket.
A bucket policy must be configured for the destination bucket, to grant the OBS system users the permission to write objects to the destination bucket. The format of the bucket policy is as follows. Replace destbucket with the actual name of the destination bucket.
```
{
	"Statement": [
		{
			"Effect": "Allow",
			"Sid": "1",
			"Principal": {"Service": "obs"},
                        "Resource": ["destbucket/*"],
			"Action": ["PutObject"]
		}
	]
}
```

Configure a bucket inventory.
We provide multiple tools to configure a bucket inventory. For details, see How to Use.

Content in an Inventory File

The content in an inventory file can be configured when creating the inventory. For details about all possible fields, see Table 1.

**Table 1** Object metadata listed in an inventory file
Metadata	Description
Bucket	Name of the source bucket
Key	The name of an object. Each object in a bucket has a unique key. (Object names in the inventory file are URL-encoded using UTF-8 character set and can be used only after being decoded.)
VersionId	Version ID of an object. If the value of IncludedObjectVersions in the inventory configuration is Current, this field is not included in the inventory file.
IsLatest	If the object version is the latest, this parameter is True. (If the value of IncludedObjectVersions in the inventory configuration is Current, this field is not included in the inventory file.)
IsDeleteMarker	When versioning is enabled for the source bucket, if an object is deleted, a new object metadata is generated for the object, and the IsDeleteMarker of the metadata is set to true. (If the value of IncludedObjectVersions in the inventory configuration is Current, this field is not included in the inventory file.)
Size	Object size, in bytes.
LastModifiedDate	Object creation date or last modification date
ETag	Hexadecimal digest of the object MD5. ETag is the unique identifier of the object content. It can be used to identify whether the object content is changed. For example, if ETag value is A when an object is uploaded and the ETag value has changed to B when the object is downloaded, it indicates that the object content is changed.
StorageClass	Storage class of an object
IsMultipartUploaded	Indicates whether an object is uploaded in the multipart mode.
ReplicationStatus	Cross-region replication status of an object
EncryptionStatus	Encryption status of an object

Inventory File Name

The name of an inventory file is in the following format:

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/files/UUID_index.csv

destinationPrefix: The inventory file name prefix configured when creating the inventory rule. Inventory files generated under the rule are named after the prefix, which can facilitate the classification of inventory files. If no prefix is specified, the default prefix is BucketInventory.
sourceBucketName: Name of the source bucket for which an inventory is configured. This field can be used to differentiate inventory files of different source buckets, if those inventory files are saved in the same destination bucket.
inventoryId: If a source bucket has multiple inventory rules whose inventory files are saved in the same destination bucket, this field can be used to identify different inventory rules.
yyyy-MM-dd'T'HH-mm'Z': Start time and date for scanning the destination bucket when an inventory file is generated. Objects uploaded to the source bucket after this time may not be listed in the inventory file.
UUID_index.csv: one of the inventory files

Manifest File

If there are a large number of objects in a bucket, multiple inventory files may be generated for a single inventory configuration. It takes some time to generate these files. For example, if there are 200,000 objects in a bucket, it takes about 1.5 minutes to generate all inventory files. One or two hours after all inventory files are generated, a manifest.json file will be generated. The manifest.json file contains information about all inventory files generated this time, including:

sourceBucket: name of the source bucket
destinationBucket: name of the destination bucket
version: version of the inventory
fileFormat: format of inventory files
fileSchema: object metadata fields contained in the inventory files
files: list of all inventory files
key: inventory file name
size: size of an inventory file, in bytes
inventoriedRecord: number of records contained in an inventory file

The following is an example of a simple manifest.json file.

{
        "sourceBucket":"user001",
        "destinationBucket":"bucket001",
        "version":"2019-01-03",
        "fileFormat":"CSV",
        "fileSchema":"Bucket,Key,Size,LastModifiedDate,ETag,StorageClass,IsMultipartUploaded,ReplicationStatus,EncryptionStatus",
        "files":[
                {
                        "key":"inventory/user001/test_id/2019-01-03T12-28Z/files/0000016813AF58E66806C1E2D7F15155_1.csv",
                        "size":6705647390,
                        "inventoriedRecord":70585762,
                }
        ]
}

The name of a manifest file is as follows (for details about each field, see Inventory File Name):

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/manifest.json

How to Use

You can configure bucket inventories using OBS Console, APIs, or SDKs.

Tool	Reference
OBS Console	Configuring a Bucket Inventory
SDKs	OBS supports software development kits (SDKs) in multiple languages. For details, see the corresponding developer guide on the SDK Overview page.
API	Configuring a Bucket Inventory