Using LTS to Parse Bucket Logs

On the LTS console, configure log parsing to map the parameters in bucket log files to bucket log fields, which helps you analyze all bucket log files.

You need to create a log group, create a log stream, configure cloud structuring parsing for the log stream, and create an index for the log stream.

LTS log parsing must be configured in the region of the log storage bucket. Otherwise, bucket logs cannot be parsed. This practice uses ME-Riyadh as an example.

1. Create a log group.

A log group is the basic unit for LTS to manage logs. It comprises log streams and categorizes them. A log group does not store any log data. It only helps with log stream management.

1. Log in to the LTS console.
2. Click in the upper part of the page and select the region where the log storage bucket is located. In this practice, select ME-Riyadh.
3. On the Log Management page, click Create Log Group.
4. On the Create Log Group page, configure the parameters as described in the following table. Retain the default settings for the other parameters.

   For more information, see Creating a Log Group.

   Table 2 Log group parameters

   Parameter	Example Value	Description
   Log Group Name	lts-group-obs-logging	A log group is the basic unit for LTS to manage logs. It is used to classify log streams. If there are too many logs to collect, separate logs into different log groups based on log types, and name log groups in an easily identifiable way. LTS automatically generates a default log group name. You are advised to customize one based on your service. You can also change it after the log group is created. The naming rules are as follows: Enter 1 to 64 characters, including only letters, digits, hyphens (-), underscores (_), and periods (.). Do not start with a period or underscore or end with a period. Each log group name must be unique.
   Enterprise Project Name	default	You can add the log group to an enterprise project for unified management. If you do not have any specific needs for enterprise project division and management, choose the default enterprise project. Enterprise projects can be used only after being enabled. For details, see Enabling Enterprise Project and Accessing Project Management.

5. Click OK.
   The created log group appears in the log group list.

   Figure 3 A log group
   

6. Hover over the log group name. Then, click next to the log group ID and note down the ID.
   Figure 4 Noting down the log group ID
   

2. Create a log stream.

LTS manages logs by log stream. After logs are collected, they are stored in different log streams based on their types for further management. Each log stream belongs to exactly one log group, while a log group can contain multiple log streams.

1. On the Log Management page, click on the left of the target log group.
2. Click Create Log Stream. On the displayed page, configure the parameters as described in the following table. Retain the default settings for the other parameters.

   For more information, see Creating a Log Stream.

   Table 3 Log stream parameters

   Parameter	Example Value	Description
   Log Stream Name	lts-topic-obs-logging	LTS automatically generates a default log stream name. You are advised to customize one based on your service. You can also change it after the log stream is created. The naming rules are as follows: Use only letters, digits, hyphens (-), underscores (_), and periods (.). Do not start with a period or underscore or end with a period. Enter 1 to 64 characters. Each log stream name must be unique.
   Enterprise Project Name	default	You can add the log stream to an enterprise project for unified management. If you do not have any specific needs for enterprise project division and management, choose the default enterprise project. Enterprise projects can be used only after being enabled. For details, see Enabling Enterprise Project and Accessing Project Management.
   Log Storage	Enable it.	Determines whether to enable log storage. With log storage enabled, logs will be stored in the search engine and all log functions are available. In this practice, do not disable log storage. Otherwise, LTS will be unable to analyze OBS bucket logs.

3. Click OK.

   The created log stream appears under the log group.

   Figure 5 A log stream
   

4. Hover over the log stream name. Then, click next to the log stream ID and note down the ID.
   Figure 6 Noting down the log stream ID
   

3. Configure cloud structuring parsing for the log stream.

Before search and analysis, you must configure structuring and indexing for the reported logs. Log structuring converts unstructured or semi-structured log data into structured data. Structured data has a unified length and format, which significantly improves the efficiency and accuracy of log search and analysis.

1. Click the name of the log stream created in 2. The log stream details page is displayed.
2. In the navigation pane, click Log Settings. On the displayed page, click the Cloud Structuring Parsing tab.
3. Configure the parameters as described in the following table. Keep the default settings for other parameters.

   For more information, see Cloud Structuring Parsing.

   

   Table 4 Cloud structuring parsing parameters

   Parameter	Example Value	Description
   Log structuring method	Nginx	Select a log structuring method based on the log content. Nginx indicates that the log_format directive is used to define custom formats for access logs.
   Step 1 Select a sample log event.	See the sample log.	Copy the contents of the OBS bucket log file into this area to use as the sample log. In this practice, copy the contents of any log file stored in the target bucket test-zcy-bucket-002 into this area.
   Step 2 Define Nginx log configuration.	See Nginx log configuration.	Define the Nginx log configuration based on the provided sample bucket log. Each defined item maps to the field name of a corresponding parameter in the sample log.
   Step 3 Extract fields.	Intelligent Extract	Select Intelligent Extract to automatically map the parameters in the sample log to the fields in the Nginx log configuration.

   • Sample log

     3c24f6f8852945a0af194f93ce087fbd test-zcy-bucket-001 [21/Jan/2026:01:42:56 +0000] 124.71.93.140 3c24f6f8852945a0af194f93ce087fbd 0d937eb40eeccf19959a28db26725d2a REST.GET.VERSIONING - "GET /test-zcy-bucket-001/?versioning HTTP/1.1" 200 - 131 - 6 6 "https://console.huaweicloud.com/obs/?locale=zh-cn&agencyId=96aab6bd80c442f3b1c1889e7ec3e2cd&region=cn-north-4" "HttpClient" - - - - "-" 96aab6bd80c442f3b1c1889e7ec3e2cd HSTAOGINYZPFDM36MT2M

   • Nginx log configuration

     log_format main '$BucketOwner $BucketName [$time_local] $RemoteIP $Requester $RequestID $Operation $Key '
     '"$Request-URI" $HTTPStatus $ErrorCode $BytesSent $ObjectSize $TotalTime $Turn-AroundTime '
     '"$Referer" "$User-Agent" $VersionID $STSLogUrn $StorageClass $TargetStorageClass '
     '"$DentryName" $IAMUserID $AccessKeyID';

4. Click Save.

4. Create an index for the log stream.

An index is a storage structure. After configuring indexes, you can query and analyze logs.

1. Click the Index Settings tab.
2. Click Auto Configure under Index Fields.

   

   Add fields extracted from structured logs to analyze statistics. Whenever the name of a field is changed on the log structuring page, the field must be re-added here.

   For more information, see Configuring Log Indexing.

3. Click OK.

Create a FunctionGraph function that obtains log files from the OBS bucket and sends them to LTS for parsing and analysis.

The FunctionGraph function must be created in the region where the log storage bucket is located. Otherwise, bucket logs cannot be obtained. This practice uses ME-Riyadh as an example.

1. Create a dependency.

A function dependency contains the common libraries required for running function code. You can encapsulate the common libraries required by code into a dependency for separate management, sharing, and smaller deployment sizes.

Function dependencies are classified into public and private dependencies. This practice uses private dependencies. A private dependency is created by locally packaging the common libraries required by the service code.

1. In the service list in the upper left corner of the console, choose Compute > FunctionGraph to go to the FunctionGraph console.
2. In the navigation pane, choose Functions > Dependencies.
3. Click Create Dependency in the upper right corner.
4. On the Create Dependency page, configure the parameters as described in the following table. Keep the default settings for other parameters.

   For more information, see Creating a Dependency.

   

   Table 5 Dependency parameters

   Parameter	Example Value	Description
   Name	esdk-obs-python	The name of the dependency. A dependency name: Can contain only letters, digits, underscores (_), periods (.), and hyphens (-). Must start with a letter and end with a letter or digit. Can be up to 96 characters long.
   Runtime	Python 3.9	The runtime language of the dependency You can select Cangjie, Node.js, Python, Java, PHP, Go, C#, or Custom.
   Code Entry Mode	Upload ZIP	Select the code upload mode. By selecting Upload ZIP, you need to upload your local code package. To do this, click Add and select the code package you downloaded from esdk-obs-python.

5. Click Create.

2. Create an agency.

Create an agency that authorizes FunctionGraph to access OBS and LTS. After obtaining log files from the OBS bucket, FunctionGraph uploads them to LTS for log analysis.

1. Log in to the Identity and Access Management console.
2. In the navigation pane, choose Agencies. Click Create Agency in the upper right corner.
   Figure 7 Creating an agency
   

3. Configure the parameters as described in the following table. Keep the default settings for other parameters.

   Table 6 Agency parameters

   Parameter	Example Value	Description
   Agency Name	OBS_LTS_2_FG_Agency	The name of the agency to be created The agency name cannot be left blank.
   Agency Type	Cloud service	Selecting Cloud service grants permissions to an account or cloud service to operate resources within the account. In this example, you need to create an agency for FunctionGraph to access LTS.
   Cloud Service	FunctionGraph	Select the cloud service you want to grant permissions to operate resources in your account. This parameter is mandatory when you set Agency Type to Cloud service. In this example, you need to create an agency for FunctionGraph to access LTS.

4. Click OK. In the Agency Created dialog box that is displayed, click Cancel. If the newly created agency appears in the agency list, it was created successfully.
5. Create a custom policy to grant LTS, KMS, and VPC permissions to the agency.
   1. In the navigation pane, choose Permissions > Policies/Roles.
   2. On the Policies/Roles page, click Create Custom Policy in the upper right corner.
   3. On the Create Custom Policy page, configure the parameters as described in the following tables. Keep the default settings for other parameters.
      Figure 8 Configuring a custom policy
      

      Table 7 Custom policy parameters

      Parameter	Example Value	Description
      Policy Name	VPC_LTS_KMS_Policy	The name of the custom policy This field cannot be left blank. Only letters (case-sensitive), digits, spaces, and special characters (-_.,) are allowed.
      Cloud service	Log Tank Service (LTS) Key Management System (KMS) Virtual Private Cloud (VPC)	Select the cloud service that the permissions apply to. You need to grant LTS, KMS, and VPC permissions to the agency. After you finish configuring permissions for one service, click Add Permissions to add permissions for the next service.
      Actions	See Table 8.	The actions that need to be granted to the agency

      Table 8 The actions that need to be granted to the agency

      Service	Action	Description
      Log Tank Service (LTS)	lts:logs:upload	Uploads logs.
      Key Management System (KMS)	kms:cmk:create	Creates a CMK.
      	kms:cmk:get	Queries key information.
      	kms:dek:create	Creates a DEK.
      	kms:cmk:list	Lists keys.
      	kms:dek:encrypt	Encrypts a DEK.
      	kms:dek:decrypt	Decrypts a DEK.
      Virtual Private Cloud (VPC)	vpc:ports:get	Queries a list of ports or the details of a port.
      	vpc:ports:create	Creates a port.
      	vpc:vpcs:get	Queries the details of a VPC.
      	vpc:subnets:get	Queries a list of subnets or the details of a subnet.

   4. Click OK. If the created custom policy appears in the list, it was created successfully.

6. Create a custom policy to grant OBS permissions to the agency.
   1. In the navigation pane, choose Permissions > Policies/Roles.
   2. On the Policies/Roles page, click Create Custom Policy in the upper right corner.
   3. On the Create Custom Policy page, configure the parameters as described in the following table. Keep the default settings for other parameters.
      Figure 9 Configuring a custom policy
      

      Table 9 Custom policy parameters

      Parameter	Example Value	Description
      Policy Name	OBS_LOG_GET_Policy	The name of the custom policy This field cannot be left blank. Only letters (case-sensitive), digits, spaces, and special characters (-_.,) are allowed.
      Cloud service	Object Storage Service (OBS)	Select the cloud service that the permissions apply to. In this practice, you need to grant the obs:object:GetObject permission to the agency.
      Actions	obs:object:GetObject	The actions that need to be granted to the agency

   4. Click OK. If the created custom policy appears in the list, it was created successfully.

7. Grant LTS, KMS, VPC, and OBS permissions to the agency.
   1. In the navigation pane, choose Agencies.
   2. Locate the agency created in 2 and click Authorize in the Operation column.
   3. On the Select Policy/Role page, select the custom policies created in 5 and 6 and click Next.
   4. On the Select Scope page, select All resources and click OK.
   5. In the Information dialog box that is displayed, click OK.
   6. On the Finish page, click Finish.

3. Create a function.

Create a function that obtains log files from the OBS bucket and uploads them to LTS for log analysis.

1. In the service list in the upper left corner of the console, choose Compute > FunctionGraph to go to the FunctionGraph console.
2. In the navigation pane, choose Functions > Function List.
3. Click Create Function.
4. On the Create Function page, configure the parameters as described in the following table. Keep the default settings for other parameters.

   For more information, see Creating an Event Function.

   

   Table 10 Function parameters

   Category	Parameter	Example Value	Description
   -	Create With	Create from scratch	Select a method for creating the function. Create from scratch means creating a custom function from the ground up.
   Basic Information	Function Type	Event Function	Defines how the function is triggered. Event Function indicates that the function is invoked by a trigger.
   	Region	ME-Riyadh	Select a region where you will deploy your function code. In this practice, select the region where the log storage bucket is located.
   	Function Name	OBS_LOG_2_LTS	The name of the function. A function name: Can contain only letters, digits, underscores (_), and hyphens (-). Must start with a letter and end with a letter or digit. Can be up to 60 characters long.
   	Enterprise Project	default	You can add the function to an enterprise project for unified management. If you do not have any specific needs for enterprise project division and management, choose the default enterprise project. For details about how to use enterprise projects, see Enterprise User Permissions and Project Management.
   	Agency	OBS_LTS_2_FG_Agency	An agency that authorizes FunctionGraph to access other cloud services. This agency must grant FunctionGraph the permissions to access LTS and OBS so that FunctionGraph can obtain log files from the OBS bucket and upload the log files to LTS for log analysis. Select the agency OBS_LTS_2_FG_Agency created in 2.
   	Runtime	Python 3.9	Select a language to compile the function.
   Advanced Settings	Public Access	Disable it.	With Public Access enabled, the function can access the Internet through the default network interface. When Public Access and VPC Access are both enabled, whether public access is available depends on the VPC configuration.
   	VPC Access	Enable it.	With VPC Access enabled, the function uses the network interface associated with the VPC for network access and disables its default network interface. Keep the default VPC and subnet. If no VPC or subnet is available, create a VPC in the same region as the function. For details, see Creating a VPC with a Subnet.
   	Collect Logs	Enable it.	Once enabled, the function's runtime logs are sent to LTS. You need to pay for log management. For details, see LTS Billing Items.
   	Log Configuration	Auto	Select the log group to be associated. If you select Auto, the default log group is associated. Do not select Custom here. Associate the log group and log stream created in Step 1: Configure Log Parsing on LTS.

5. Click Create Function. The function creation is complete, and the function details page is displayed.
6. Copy the following code to the code area on the Code tab. Change the values of obs_endpoint and lts_ip, and click Deploy.
   • Change the value of obs_endpoint to the endpoint of the log storage bucket. In this practice, the value is obs.me-east-1.myhuaweicloud.com. You can obtain the endpoint from the Domain Name Details area on the bucket's Overview page. For details, see Viewing Bucket Information.
   • Change the region ID in the value of lts_ip to the ID of the region where the log storage bucket is located. In this practice, the region ID is me-east-1. To obtain the region ID, see Regions and Endpoints.

     import time
     import urllib.parse
     import os
     import random
     import string
     import json
     import requests
     from obs import ObsClient
     
     localMountPath = '/tmp/'
     
     
     def handler(event, context):
         log_group_id = context.getUserData("log_group_id")
         log_stream_id = context.getUserData("log_stream_id")
     
         logger = context.getLogger()
         auth_token = context.getToken()
         project_id = context.getProjectID()
         obs_endpoint = "obs.me-east-1.myhuaweicloud.com"
         lts_ip = "https://lts-access.me-east-1.myhuaweicloud.com:8102"
     
         data = event.get("data", None)
         (src_bucket, input_object_key) = get_obs_obj_info(data)
         logger.info("src bucket name: %s", src_bucket)
     
         input_object_key = urllib.parse.unquote_plus(input_object_key)
         logger.info("src object key: %s", input_object_key)
     
         download_dir = gen_local_download_path()
         obs_client = new_obs_client(context, obs_endpoint)
         if not download_from_obs(context, obs_client, src_bucket, input_object_key, download_dir):
             return 'ERROR'
     
         url = lts_ip + '/v2/' + project_id + '/lts/groups/' + log_group_id + '/streams/' + log_stream_id + '/tenant/contents'
         headers = {'content-type': 'application/json', 'X-Auth-Token': auth_token}
     
         download_file = download_dir + input_object_key
         f = open(download_file)
         lines = f.readlines()
         for line in lines:
             logger.info(line)
             payload = {"log_time_ns": int(round(time.time() * 1000000000)), "contents": line,
                        "labels": {"user_tag": "string"}}
             response = requests.post(url, data=json.dumps(payload), headers=headers, verify=False).text
             logger.info(response)
     
         os.remove(download_file)
         return 'SUCCESS'
     
     
     def download_from_obs(context, obs_client, bucket, object_key, download_dir):
         resp = obs_client.getObject(bucket, object_key, downloadPath=download_dir + object_key)
         if resp.status < 300:
             return True
         else:
             context.getLogger().error('failed to download_from_obs '
                                       'errorCode: %s, errorMessage: %s' % (
                                           resp.errorCode, resp.errorMessage))
             return False
     
     
     def gen_local_download_path():
         download_path = localMountPath
     
         if not download_path.endswith('/'):
             download_path += '/'
     
         letters = string.ascii_letters
         download_dir = download_path + ''.join(
             random.choice(letters) for i in range(16)) + '/'
         os.makedirs(download_dir)
         return download_dir
     
     
     def new_obs_client(context, obsServer):
         ak = context.getSecurityAccessKey()
         sk = context.getSecuritySecretKey()
         st = context.getSecurityToken()
         return ObsClient(access_key_id=ak, secret_access_key=sk, security_token=st, server=obsServer)
     
     
     def get_obs_obj_info(record):
         if 's3' in record:
             s3 = record['s3']
             return (s3['bucket']['name'], s3['object']['key'])
         else:
             obs_info = record['obs']
             return (obs_info['bucket']['name'], obs_info['object']['key'])

7. Add code dependencies. The sample code in 6 depends on the OBS SDK, which must be added as a dependency. Follow the following steps to add it:
   1. At the bottom of the Code tab, click Add under Dependencies.
   2. In the Select Dependency dialog box on the right, set Type to Private.
   3. For Name, select the dependency esdk-obs-python created in 3.
   4. Click OK.

8. Configure the basic settings of the function.
   1. In the middle of the function details page, choose Configuration > Basic Settings.
   2. Configure the following parameters:
      • Execution Timeout (s): 10000
      • Memory (MB): 1024
   3. Click Save.

9. Configure environment variables. They are used to associate the LTS log group and log stream.
   1. On the Configuration tab, choose Environment Variables in the navigation pane.
   2. Click Edit Environment Variable. In the dialog box displayed on the right, click Add.
   3. Add the environment variables listed in the following table and click OK.

      Table 11 Environment variables

      Key	Value	Description
      log_group_id	714a4f22-8c85-4a43-a8d1-9b60fe64004a	The ID of an LTS log group The value is the ID of the log group created in Step 1: Configure Log Parsing on LTS.
      log_stream_id	c7a4f097-bfbb-44ee-b33f-30a0dede1e87	The ID of an LTS log stream The value is the ID of the log stream created in Step 1: Configure Log Parsing on LTS.

Use an EG notification as the event trigger to trigger function execution. An EG event subscription connects an event source, a channel, and an event target. The event source (an OBS event) uses the channel to trigger the event target (the function) to run.

The EG event subscription must be created in the region where the log storage bucket is located. Otherwise, the event source (OBS event) cannot be obtained. This practice uses ME-Riyadh as an example.

1. Click the service list in the upper left corner of the page. Then, choose Storage > Object Storage Service. The bucket list page is displayed.
2. Click the name of the log storage bucket. The object list page is displayed.
3. In the navigation pane, choose Data Management > Event Notification.
4. On the EG tab, click Create. The EG event subscription details page is displayed.
5. Change the subscription name.

   Click next to the subscription name in the upper right corner. In the displayed dialog box, change the subscription name and click OK. In this example, change the subscription name to OBS_LTS_Subscription.

6. Configure an event source.
   1. Click Event Source. In the displayed dialog box, configure the parameters as described in the following table. Keep the default settings for other parameters.

      For more information, see Creating an Event Subscription.

      Figure 10 Configuring an event source
      

      Table 12 Event source parameters

      Parameter	Example Value	Description
      Provider	Cloud services	Select an event source provider. Cloud services indicates that Huawei Cloud services are the event source providers.
      Event Source	OBS Application Service	Select a cloud service event source. OBS Application Service indicates that certain operations on objects in the bucket will generate events.
      Bucket	test-zcy-bucket-002	Select the OBS bucket (the log storage bucket in this example) for which events are generated.
      Event Type	OBS:DWR:ObjectCreated:PUT OBS:DWR:ObjectCreated:POST	There will be events when the following operations happen: OBS:DWR:ObjectCreated:PUT: Creates or overwrites objects using the console or Put requests. OBS:DWR:ObjectCreated:POST: Creates or overwrites objects using Post requests.
      Object Name Prefix	test-zcy-bucket-001-log/	Only the OBS events generated by the objects with the specified prefix can trigger EG notifications. Enter the directory used to store logs in the OBS log storage bucket.

   2. Click OK.

7. Configure an event target.
   1. Click Event Target. In the displayed dialog box, configure the parameters as described in the following table. Keep the default settings for other parameters.

      For more information, see Creating an Event Subscription.

      Figure 11 Configuring an event target
      

      Table 13 Event target parameters

      Parameter	Example Value	Description
      Provider	Cloud services	Select an event target provider. Cloud services indicates that Huawei Cloud services are the event target providers.
      Event Target	FunctionGraph (function computing)	Select a cloud service event target. FunctionGraph (function computing) indicates that FunctionGraph executes specific operations.
      Function	OBS_LOG_2_LTS	Select the required function. In this practice, select the function created in Step 2: Create a Function on FunctionGraph.
      Agency	EG_TARGET_AGENCY	An agency that authorizes EG to access other cloud services. In this practice, the agency needs to authorize EG to access FunctionGraph. If no agency is available, click Create Agency to automatically create one named EG_TARGET_AGENCY.

   2. Click OK.

8. Click Save in the upper right corner of the page.

1. Log in to the LTS console.
2. Click in the upper part of the page and select the region where the log group (created in Step 1: Configure Log Parsing on LTS) is located. In this practice, select ME-Riyadh.
3. In the middle of the Log Management page, click on the left of the log group created in Step 1: Configure Log Parsing on LTS and click the log stream name.

   

4. On the Log Search page of the log stream, check the parsed bucket logs.
   Figure 12 Checking the log analysis result
   

   You can select fields in the left pane to check logs of specific field values. For example, you can choose ErrorCode > 404 to filter 404 access logs. For more information about log search and analysis, see Searching and Analyzing Logs.