Using Logging to Record OBS Logs

Updated on 2024-10-24 GMT+08:00

After logging is enabled for a bucket, OBS automatically converts bucket logs into objects following the naming rules and writes the objects into a target bucket.

Scenarios

You can enable logging to facilitate analysis or audit. Access logs enable a bucket owner to analyze the property, type, or trend of requests to the bucket in depth. When the logging function of a bucket is enabled, OBS will log access requests for the bucket automatically, and write the generated log files to the specified bucket (target bucket).

You need to specify a bucket for storing log files when enabling logging for a bucket. Log files can be stored in any bucket in the region where the logged bucket resides, including the logged bucket itself. To better manage logs, you are advised to store log files in a bucket other than the logged bucket. If log files are stored in the logged bucket, OBS creates additional logs for writing log files to the bucket, which takes up extra storage space that will increase your costs and makes it more difficult for you to locate required logs.

NOTICE:

Uploading bucket logs to the target bucket incurs billable PUT requests. For details about the pricing, see Requests.

After logging is enabled, the log delivery user group will be automatically granted the permission to read the bucket ACL and write the bucket where logs are saved. If you manually disable such permissions, bucket logging will fail.

OBS can log bucket access requests for further request analysis or log audit.

Logs occupy the OBS storage that incurs costs, so OBS does not collect bucket access logs by default.

OBS creates log files and uploads them to a specified bucket. To perform these operations, OBS must be granted required permissions. Therefore, before configuring logging for a bucket, you need to create an IAM agency for OBS and add this agency when configuring logging for the bucket. By default, when configuring permissions for an agency, you only need to grant the agency the permission to upload log files (PutObject) to the bucket for storing log files. In the following example, mybucketlogs is the bucket. If the log storage bucket has server-side encryption enabled, the agency also requires the KMS Administrator permission for the region where the bucket is located.

{
    "Version": "1.1",
    "Statement": [
        {
            "Action": [
                "obs:object:PutObject"
            ],
            "Resource": [
                "OBS:*:*:object:mybucketlogs/*"
            ],
            "Effect": "Allow"
        }
    ]
}

After logging is configured, you can view operation logs in the bucket that stores the logs in approximately fifteen minutes.

The following shows an example access log of the target bucket:

787f2f92b20943998a4fe2ab75eb09b8 bucket [13/Aug/2015:01:43:42 +0000] xx.xx.xx.xx 
787f2f92b20943998a4fe2ab75eb09b8 281599BACAD9376ECE141B842B94535B  REST.GET.BUCKET.LOCATION 
- "GET /bucket?location HTTP/1.1" 200 - 211 - 6 6 "-"  "HttpClient" - -

The access log of each bucket contains the following information.

**Table 1** Bucket log format
Name	Example	Description
BucketOwner	787f2f92b20943998a4fe2ab75eb09b8	Account ID of the bucket owner
Bucket	bucket	Name of the bucket
Time	[13/Aug/2015:01:43:42 +0000]	Timestamp of the request (UTC)
Remote IP	xx.xx.xx.xx	IP address from where the request is initiated
Requester	787f2f92b20943998a4fe2ab75eb09b8	Requester ID When an account initiates a request, this parameter value is the account ID. When an IAM user initiates a request, this parameter value is the ID of the account where the IAM user belongs. When an anonymous user initiates a request, this parameter value is Anonymous.
RequestID	281599BACAD9376ECE141B842B94535B	Request ID
Operation	REST.GET.BUCKET.LOCATION	Name of the operation See Table 2 for common operations and their description.
Key	-	Object name
Request-URI	GET /bucket?location HTTP/1.1	URI of the request
HTTPStatus	200	Response code
ErrorCode	-	Error code
BytesSent	211	Size of the HTTP response, expressed in bytes
ObjectSize	-	Object size (bytes) NOTE: When OBS deletes an object, it does not log the object's size. In the object deletion log, the value of ObjectSize is 0. If error code 4XX is returned, the value of ObjectSize is -, indicating that the specific object size is not displayed.
TotalTime	6	Processing time on the server (ms)
Turn-AroundTime	6	Total time for processing the request (ms) NOTE: This parameter can also be written as TotalTime.
Referer	-	Header field Referer of the request
User-Agent	HttpClient	User-Agent header of the request
VersionID	-	Version ID carried in the request
STSLogUrn	-	Federated authentication and agency information
StorageClass	STANDARD_IA	Current storage class of the object
TargetStorageClass	GLACIER	Storage class that the object will be transited to
DentryName	12456/file.txt	For a parallel file system, this field indicates an internal identifier of a file or directory. Its value consists of a parent directory inode number and a file or directory name. For a bucket, the value of this field is -.
IAMUserID	8f3b8c53d29244a780084f2b8c106c32	ID of an IAM user. When a request is initiated by an anonymous user, Anonymous is logged.

**Table 2** Common operations
Operation	Description	Operation	Description
REST.GET.SERVICE	Lists buckets.	REST.GET.ENCRYPTION	Obtains the bucket encryption configuration.
REST.PUT.BUCKET	Creates a bucket.	REST.DELETE.ENCRYPTION	Deletes the bucket encryption configuration.
REST.HEAD.BUCKET	Views the bucket information.	REST.PUT.OTM_DIRECT_COLD_ACCESS	Configures direct reading for Archive objects in a bucket.
REST.GET.BUCKETVERSIONS	Lists objects in a bucket.	REST.GET.OTM_DIRECT_COLD_ACCESS	Obtains the direct reading configuration of a bucket.
REST.GET.BUCKET	Obtains the bucket metadata.	REST.DELETE.OTM_DIRECT_COLD_ACCESS	Deletes the direct reading configuration of a bucket.
REST.GET.BUCKET.LOCATION	Obtains the bucket location.	REST.PUT.BUCKET.WEBSITE	Configures static website hosting for a bucket.
REST.DELETE.BUCKET	Deletes a bucket.	REST.GET.BUCKET.WEBSITE	Obtains the static website hosting configuration of a bucket.
REST.PUT.POLICY	Configures a bucket policy.	REST.DEL.BUCKET.WEBSITE	Deletes the static website hosting configuration of a bucket.
REST.GET.POLICY	Obtains a bucket policy.	REST.PUT.BUCKET.CORS	Configures CORS for a bucket.
REST.DELETE.POLICY	Deletes a bucket policy.	REST.GET.BUCKET.CORS	Obtains the CORS configuration of a bucket.
REST.PUT.ACL	Configures an ACL for a bucket or an object.	REST.DEL.BUCKET.CORS	Deletes the CORS configuration of a bucket.
REST.GET.ACL	Obtains a bucket ACL or an object ACL.	REST.OPTIONS.BUCKET	Checks bucket OPTIONS.
REST.PUT.LOGGING_STATUS	Configures logging for a bucket.	REST.OPTIONS.OBJECT	Checks object OPTIONS.
REST.GET.LOGGING_STATUS	Obtains the bucket logging configuration.	REST.PUT.OBJECT	Uploads an object with PUT.
REST.PUT.BUCKET.LIFECYCLE	Configures a bucket lifecycle rule.	REST.POST.OBJECT	Uploads an object with POST.
REST.GET.LIFECYCLE	Obtains the lifecycle configuration of a bucket.	REST.COPY.OBJECT	Copies an object.
REST.DEL.LIFECYCLE	Deletes the lifecycle configuration of a bucket.	REST.GET.OBJECT	Obtains the object content.
REST.PUT.VERSIONING	Configures versioning for a bucket.	REST.HEAD.OBJECT	Obtains the object metadata.
REST.GET.VERSIONING	Obtains the bucket versioning status.	REST.DELETE.OBJECT	Deletes an object.
REST.GET.BUCKET.STORAGE.POLICY	Configures the default storage class for a bucket.	REST.TRANSITION.STORAGECLASS.OBJECT	Changes the storage class of an object.
REST.PUT.BUCKET.STORAGE.POLICY	Obtains the default storage class of a bucket.	OP_MULTIPLE_DELETEOBJECT	Batch deletes objects.
REST.PUT.REPLICATION	Configures cross-region replication for a bucket.	REST.POST.RESTORE	Restores an Archive object.
REST.DELETE.REPLICATION	Deletes the cross-region replication configuration of a bucket.	REST.APPEND.OBJECT	Appends data to an object.
REST.GET.REPLICATION	Obtains the cross-region replication configuration of a bucket.	REST.MODIFY.OBJECT.META	Modifies object metadata.
REST.PUT.TAGGING	Configures tags for a bucket.	REST.TRUNCATE.OBJECT	Truncates an object.
REST.GET.TAGGING	Obtains bucket tags.	REST.RENAME.OBJECT	Renames an object.
REST.DEL.TAGGING	Deletes bucket tags.	REST.GET.UPLOADS	Lists the initiated multipart uploads in a bucket.
REST.PUT.BUCKET_QUOTA	Configures a storage quota for a bucket.	REST.POST.UPLOADS	Initiates a multipart upload.
REST.GET.BUCKET.QUOTA	Queries the bucket storage quota.	REST.PUT.PART	Uploads a part.
REST.GET.BUCKET.STORAGEINFO	Queries the information about the used space in a bucket.	REST.COPY.PART	Copies a part.
REST.PUT.BUCKET.INVENTORY	Configures inventories for a bucket.	REST.GET.UPLOAD	Lists uploaded parts.
REST.GET.BUCKET.INVENTORY	Obtains or lists bucket inventories.	REST.POST.UPLOAD	Assembles parts.
REST.DELETE.BUCKET.INVENTORY	Deletes bucket inventories.	REST.DELETE.UPLOAD	Aborts a multipart upload.
REST.PUT.CUSTOMDOMAIN	Configures a custom domain name for a bucket.	REST.CLEAR.EXPIRE.UPLOAD	Deletes expired segments.
REST.GET.CUSTOMDOMAIN	Obtains the custom domain name of a bucket.	REST.DELETE.CUSTOMDOMAIN	Deletes a custom domain name of a bucket.
REST.PUT.ENCRYPTION	Configures encryption for a bucket.	-	-

Ways to Configure Logging for a Bucket

You can use OBS Console, APIs, or SDKs to configure logging for a bucket.

Using OBS Console

In the navigation pane of OBS Console, choose Object Storage.
In the bucket list, click the bucket you want to operate. The Objects page is displayed.
In the navigation pane, choose Overview.
In the Basic Configurations area, click Logging. The Logging dialog box is displayed.
Select Enable. For details, see Figure 1.

Figure 1 Logging
Select an existing bucket where you want to store log files. Log delivery users of the selected bucket will be automatically granted the permissions to read the bucket ACL and write logs to the bucket.
Enter a prefix for the Log File Name Prefix.

After logging is enabled, generated logs are named in the following format:

<Log File Name Prefix>YYYY-mm-DD-HH-MM-SS-<UniqueString>
- <Log File Name Prefix> is the shared prefix of log file names.
- YYYY-mm-DD-HH-MM-SS indicates when the log is generated.
- <UniqueString> indicates a character string generated by OBS.
On OBS Console, if the configured <Log File Name Prefix> ends with a slash (/), logs generated in the bucket are stored in the <Log File Name Prefix> folder in the bucket, facilitating the management of log files.

Example:
- If the bucket named bucket is used to save log files, and the log file name prefix is set to bucket-log/, all log files delivered to this bucket are saved in the bucket-log folder. A log file is named as follows: 2015-06-29-12-22-07-N7MXLAF1BDG7MPDV.
- If the bucket named bucket is used to save log files, and the log file name prefix is set to bucket-log, all log files are saved in the root directory of the bucket. A log file is named as follows: bucket-log2015-06-29-12-22-07-N7MXLAF1BDG7MPDV.
Select an IAM agency to grant OBS the permission to upload log files to the specified bucket.

By default, you only need to grant the agency the permission to upload log files (PutObject) to the log storage bucket (mybucketlogs as an example). If the log storage bucket has Server-Side Encryption enabled, the agency also requires the KMS Administrator permission for the region where the bucket is located.
```
{
    "Version": "1.1",
    "Statement": [
        {
            "Action": [
                "obs:object:PutObject"
            ],
            "Resource": [
                "OBS:*:*:object:mybucketlogs/*"
            ],
            "Effect": "Allow"
        }
    ]
}
```
You can choose an existing IAM agency from the drop-down list or click Create Agency to create one. For details about how to create an agency, see Creating an Agency for Uploading Logs.
Click OK.

After logging is configured, you can view operation logs in the bucket that stores the logs in approximately fifteen minutes.

Using the API

Configuring Logging for a Bucket

Using SDKs

Related Operations

Disable bucket logging.

If you no longer need to record logs, in the Logging dialog box, click Disable and then click OK. After logging is disabled, logs are not recorded, but existing logs in the target bucket will be retained.

Configure an agency for uploading logs.

Creating an Agency for Uploading Logs

In the Logging dialog box, click Create Agency to jump to the Agencies page on the Identity and Access Management console.
Click Create Agency.
Enter an agency name.
Select Cloud service for the Agency Type.
Select Object Storage Service (OBS) for Cloud Service.
Set a validity period.
Click Next.
On the Select Policy/Role page, select a custom policy that has the permission to upload data to the log storage bucket and click Next.

If no custom policy is available, create one by referring to Creating a Custom Policy.

Select Global services for Scope. Select JSON for Policy View. The policy content is as follows.

NOTE:

When coding the policy content in an actual scenario, replace mybucketlogs with the actual bucket name:
```
{
    "Version": "1.1",
    "Statement": [
        {
            "Action": [
                "obs:object:PutObject"
            ],
            "Resource": [
                "OBS:*:*:object:mybucketlogs/*"
            ],
            "Effect": "Allow"
        }
    ]
}
```
On the Select Scope page, select Global services for Scope and click OK.
(Optional) If the log storage bucket has server-side encryption enabled, the agency also requires the KMS Administrator permission for the region where the bucket is located.
1. Go to the Agencies page of the IAM console and click the name of the agency created in the previous step.
2. Choose the Permissions tab and click Authorize.
3. On the Select Policy/Role page, search for and select KMS Administrator. Then, click Next.
4. On the Select Scope page, select Region-specific projects for Scope. Then, select the project in the region where the log storage bucket is located.