Bucket Inventories

Scenarios

A bucket inventory can list objects in a bucket, save the related object information in CSV files, and deliver the CSV files to the bucket specified for storing bucket inventory files. In this manner, you can easily manage objects in a bucket. A source bucket can also be the destination bucket.

• A bucket inventory file can contain the following object related information: versions, sizes, storage classes, tags, encryption statuses, and last modification time.
• You can encrypt bucket inventory files in the SSE-KMS mode.
• You can set the frequency (daily or weekly) for generating bucket inventory files.
• You can also specify a bucket to store the generated bucket inventory files.

Constraints

Bucket versions

• Inventories can be generated only for OBS 3.0 buckets, but they can be stored in either OBS 3.0 or OBS 2.0 buckets.

Number of bucket inventories

• A bucket can have a maximum of 10 inventories.

Source and destination buckets

• The source bucket (for which a bucket inventory rule is configured) and the destination bucket (where the generated inventory files are stored) must belong to the same account.
• The source and destination buckets must be in the same region.
• The destination bucket cannot have server-side encryption enabled.

Functions

• Inventory files must be in the CSV format.
• Inventories can apply to all objects in a bucket or a set of objects with the same name prefix.
• Inventory rules in the same bucket cannot overlap.
  • If there is already an inventory rule for all objects in the bucket, any other inventory rule with an object name prefix specified cannot be created. To create a rule for only a set of objects, first delete the inventory rule configured for all objects.
  • As long as there is an inventory rule for a set of objects, a rule for all objects in the bucket cannot be created. To create a rule for all objects, first delete all inventory rules that match objects by prefix.
  • If a bucket already has an inventory rule that filters objects by the object name prefix ab, the filter of a new inventory rule cannot start with a or abc. To create such a rule, you need to first delete the existing inventory rule that conflicts with the rule you will create.
• Only SSE-KMS can be used to encrypt bucket inventories.

Permissions

• Inventory files are uploaded to the destination bucket by an OBS system user, so you must grant this user the write permission for the bucket. That is, the destination bucket must contain a policy with the value of {"Service": "obs"} for Principal. For details, see 1.

Others

• The bucket inventory function is offered for free, but inventory files are billed for the storage space they occupy.
• The bucket inventory function is not available for federated users.

Configuring a Bucket Inventory

Configuring a Bucket Inventory

You use OBS Console or call the API to configure a bucket inventory. If you configure a bucket inventory on OBS Console, a bucket policy with the required permission configuration is automatically generated for the destination bucket. If you call the API to configure the bucket inventory, you need to manually configure the bucket policy for the destination bucket.

1. Add a bucket policy for the destination bucket.

   A bucket policy must be configured for the destination bucket, to grant the OBS system users the permission to write objects to the destination bucket. The format of the bucket policy is as follows. Replace destbucket with the actual name of the destination bucket.

   {
   	"Statement": [
   		{
   			"Effect": "Allow",
   			"Sid": "1",
   			"Principal": {"Service": "obs"},
                           "Resource": ["destbucket/*"],
   			"Action": ["PutObject"]
   		}
   	]
   }

2. Configure a bucket inventory.

   We provide multiple tools to configure a bucket inventory. For details, see Bucket Inventories.

Content in an Inventory File

The content in an inventory file can be configured when creating the inventory. For details about all possible fields, see Table 1.

Inventory File Name

The name of an inventory file is in the following format:

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/files/UUID_index.csv

• destinationPrefix: The inventory file name prefix configured when creating the inventory rule. Inventory files generated under the rule are named after the prefix, which can facilitate the classification of inventory files. If no prefix is specified, the default prefix is BucketInventory.
• sourceBucketName: Name of the source bucket for which an inventory is configured. This field can be used to differentiate inventory files of different source buckets, if those inventory files are saved in the same destination bucket.
• inventoryId: If a source bucket has multiple inventory rules whose inventory files are saved in the same destination bucket, this field can be used to identify different inventory rules.
• yyyy-MM-dd'T'HH-mm'Z': Start time and date for scanning the destination bucket when an inventory file is generated. Objects uploaded to the source bucket after this time may not be listed in the inventory file.
• UUID_index.csv: one of the inventory files

Manifest File

If there are a large number of objects in a bucket, multiple inventory files may be generated for a single inventory configuration. It takes some time to generate these files. For example, if there are 200,000 objects in a bucket, it takes about 1.5 minutes to generate all inventory files. One or two hours after all inventory files are generated, a manifest.json file will be generated. The manifest.json file contains information about all inventory files generated this time, including:

• sourceBucket: name of the source bucket
• destinationBucket: name of the destination bucket
• version: version of the inventory
• fileFormat: format of inventory files
• fileSchema: object metadata fields contained in the inventory files
• files: list of all inventory files
• key: inventory file name
• size: size of an inventory file, in bytes
• inventoriedRecord: number of records contained in an inventory file

{
        "sourceBucket":"user001",
        "destinationBucket":"bucket001",
        "version":"2019-01-03",
        "fileFormat":"CSV",
        "fileSchema":"Bucket,Key,Size,LastModifiedDate,ETag,StorageClass,IsMultipartUploaded,ReplicationStatus,EncryptionStatus",
        "files":[
                {
                        "key":"inventory/user001/test_id/2019-01-03T12-28Z/files/0000016813AF58E66806C1E2D7F15155_1.csv",
                        "size":6705647390,
                        "inventoriedRecord":70585762,
                }
        ]
}

The name of a manifest file is as follows (for details about each field, see Inventory File Name):

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/manifest.json

symlink.txt File

The symlink.txt file records the path of an inventory file. It helps quickly find all inventory files in big data scenarios. Apache Hive is compatible with the symlink.txt file. Hive can automatically find the symlink.txt file and the inventory files recorded in it.

The name of the symlink.txt file is as follows (for details about each field, see Inventory File Name):

destinationPrefix/sourceBucketName/inventoryId/hive/dt=YYYY-MM-DD-00-00/symlink.txt

Ways to Configure a Bucket Inventory

You can use OBS Console, APIs, or SDKs to configure a bucket inventory.