Bucket Inventory

Scenarios

A bucket inventory can list objects in a bucket, save the related object information in CSV files, and deliver the CSV files to the bucket specified for storing bucket inventory files. In this manner, you can easily manage objects in a bucket. A source bucket can also be the destination bucket.

• A bucket inventory file can contain the following object related information: versions, sizes, storage classes, tags, encryption statuses, and last modification time.
• You can encrypt bucket inventory files in the SSE-KMS mode.
• You can set the frequency (daily or weekly) for generating bucket inventory files.
• You can also specify a bucket to store the generated bucket inventory files.

Constraints

Bucket versions

• Inventories can be generated only for OBS 3.0 buckets, but they can be stored in either OBS 3.0 or OBS 2.0 buckets.

Number of bucket inventories

• A bucket can have a maximum of 10 inventories.

Source and destination buckets

• The source bucket (for which a bucket inventory rule is configured) and the destination bucket (where the generated inventory files are stored) must belong to the same account.
• The source bucket and the destination bucket must be in the same region.
• Default encryption cannot be enabled for the destination bucket configured for storing inventory files.

Functions

• Inventory files must be in the CSV format.
• OBS can generate inventory files for all objects in a bucket or a group of objects whose names begin with the same prefix.
• If a bucket has multiple inventory rules, these rules must not overlap.
  • If a bucket already has an inventory rule for the entire bucket, new inventory rules that filter objects by prefixes cannot be created. If you need an inventory rule that covers only a subset of objects in the bucket, first delete the inventory rule configured for the entire bucket.
  • If an inventory rule that filters objects by a specified prefix already exists, you cannot create an inventory rule for the entire bucket. To create an inventory rule for the entire bucket, make sure that the bucket has no other inventory rules that filter objects by specified prefixes.
  • If a bucket already has an inventory rule that filters objects by the object name prefix ab, the filter of a new inventory rule cannot start with a or abc. Or, you can delete the existing inventory rule and create a new one that filters objects according to your needs.
• Only SSE-KMS can be used to encrypt bucket inventories.

Permissions

• Inventory files are delivered to the destination bucket by an OBS system user. Therefore, you need to authorize the system user the permission to write the destination bucket.

How Is a Bucket Inventory Configured?

Configuring a Bucket Inventory

You use OBS Console or call the API to configure a bucket inventory. If you configure a bucket inventory on OBS Console, a bucket policy with the required permission configuration is automatically generated for the destination bucket. If you call the API to configure the bucket inventory, you need to manually configure the bucket policy for the destination bucket.

1. Add a bucket policy for the destination bucket.

   A bucket policy must be configured for the destination bucket, to grant the OBS system users the permission to write objects to the destination bucket. The format of the bucket policy is as follows. Replace destbucket with the actual name of the destination bucket.

   {
   	"Statement": [
   		{
   			"Effect": "Allow",
   			"Sid": "1",
   			"Principal": {"Service": "obs"},
                           "Resource": ["destbucket/*"],
   			"Action": ["PutObject"]
   		}
   	]
   }

2. Configure a bucket inventory.

   We provide multiple tools to configure a bucket inventory. For details, see How to Use.

Content in an Inventory File

The content in an inventory file can be configured when creating the inventory. For details about all possible fields, see Table 1.

Inventory File Name

The name of an inventory file is in the following format:

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/files/UUID_index.csv

Manifest File

If there are a large number of objects in a bucket, multiple inventory files may be generated for a single inventory configuration. It takes some time to generate these files. For example, if there are 200,000 objects in a bucket, it takes about 1.5 minutes to generate all inventory files. One or two hours after all inventory files are generated, a manifest.json file will be generated. The manifest.json file contains information about all inventory files generated this time, including:

• sourceBucket: name of the source bucket
• destinationBucket: name of the destination bucket
• version: version of the inventory
• fileFormat: format of inventory files
• fileSchema: object metadata fields contained in the inventory files
• files: list of all inventory files
• key: inventory file name
• size: size of an inventory file, in bytes
• inventoriedRecord: number of records contained in an inventory file

{
        "sourceBucket":"user001",
        "destinationBucket":"bucket001",
        "version":"2019-01-03",
        "fileFormat":"CSV",
        "fileSchema":"Bucket,Key,Size,LastModifiedDate,ETag,StorageClass,IsMultipartUploaded,ReplicationStatus,EncryptionStatus",
        "files":[
                {
                        "key":"inventory/user001/test_id/2019-01-03T12-28Z/files/0000016813AF58E66806C1E2D7F15155_1.csv",
                        "size":6705647390,
                        "inventoriedRecord":70585762,
                }
        ]
}

The name of a manifest file is as follows (for details about each field, see Inventory File Name):

destinationPrefix/sourceBucketName/inventoryId/yyyy-MM-dd'T'HH-mm'Z'/manifest.json

How Is a User Notified When Inventory Files Are Generated?

You can enable SMN (the message notification service) for the destination bucket. By doing so, you can receive SMS messages or emails every time when inventory files and the manifest file are generated. For more information about SMN, see Event Notifications.

The following is a simple example of SMN configuration. destinationPrefix/sourceBucketName indicates the prefix of the manifest file. destinationPrefix is the configured name prefix for inventory files, and sourceBucketName is the source bucket for which the inventory file is configured. The manifest.json is the suffix of the manifest file.

<NotificationConfiguration>
  <TopicConfiguration>
    <Id>01</Id>
    <Filter>
      <Object>
        <FilterRule>
          <Name>prefix</Name>
          <Value>destination-prefix/source-bucket</Value>
        </FilterRule>
        <FilterRule>
          <Name>suffix</Name>
          <Value>manifest.json</Value>
        </FilterRule>
      </Object>
    </Filter>
    <Topic>urn:smn:southchina:11aa22bb:notification</Topic>
    <Event>ObjectCreated:Put</Event>
  </TopicConfiguration>
</NotificationConfiguration>

How to Use

You can configure bucket inventories using OBS Console.