Setting Indexes

Index Types

LTS supports full-text and field indexes. For details, see Table 1.

Precautions

• Either whole text indexing or index fields must be configured.
• Creating an index generates index traffic and occupies index storage space. For details about the fee, see Price Calculator.
• After the index function is disabled, the storage space of historical indexes is automatically cleared after the data storage period of the current log stream expires.
• LTS enables index fields for certain system reserved fields by default. For details, see System Reserved Fields.
• Different index settings will generate different query and analysis results. Configure the index settings as required. Full-text indexes and index fields do not affect each other.
• After the index configuration is modified, the modification takes effect only for newly written log data.
• Before the field index function is available, fields supported by SQL analysis come from structured parsing on the cloud. After the field index function is available, fields supported by SQL analysis come from the field index as long as the field index is configured. Therefore, modifying the field index may affect the query results in the existing visualized charts, dashboards, SQL alarms, scheduled SQL statements, and Grafana ingestion. Exercise caution when performing this operation.
• When the field index is set to the JSON data type, quick analysis, chart redirection, and visualization on the Log Search tab page are not supported for JSON subfields. Instead, quick analysis and SQL visualization on the Search Analysis page are supported for JSON subfields.
• When the field index is set to the JSON data type, the hit result of querying JSON parent fields can be highlighted, but the hit result of querying JSON subfields cannot be highlighted.

Configuring Full-Text Indexing

1. Log in to the LTS console. The Log Management page is displayed by default.
2. Expand the target log group and click the name of the target stream.
3. On the log stream details page, click in the upper right corner. On the page displayed, click the Index Settings tab.
4. Set parameters by referring to Table 2. Index Whole Text is enabled by default.
   

   • For automatic configuration, LTS obtains the intersection of the raw logs and system fields in the last 15 minutes by default, and combines the intersection with current structured fields and tag fields to form the table data below Index Fields.
   • If no raw log is generated in the last 15 minutes, LTS obtains hostIP, hostName, pathFile, structured fields, and tag fields to form the table data.
   • When structuring is configured for ECS ingestion, the category, hostName, hostId, hostIP, hostIPv6 and pathFile fields are automatically added to Index Fields on the Index Settings page. A field will not be added if the same one already exists.
   • When structuring is configured for CCE ingestion, the category, clusterId, clusterName, nameSpace, podName, containerName, appName, hostName, hostId, hostIP, hostIPv6 and pathFile fields are automatically added to Index Fields on the Index Settings page. A field will not be added if the same one already exists.

   Table 2 Custom full-text indexing parameters

   Parameter	Description
   Index Whole Text	If Index Whole Text is enabled, a full-text index is created.
   Case-Sensitive	Indicates whether letters are case-sensitive during query. If this function is enabled, the query result is case-sensitive. For example, if the example log contains Know, you can query the log only with Know. If this function is disabled, the query result is case-insensitive. For example, if the example log contains Know, you can also query the log with KNOW or know.
   Include Chinese	Indicates whether to distinguish between Chinese and English during query. After the function is enabled, if the log contains Chinese characters, the Chinese content is split based on unigram segmentation and the English content is split based on delimiters. NOTE: Unigram segmentation is to split a Chinese string into Chinese characters. The advantage of unigram segmentation is efficient word segmentation of massive logs, and other Chinese segmentation methods have great impact on the write speed. If Include Chinese is enabled, unigram segmentation is used for Chinese characters (each Chinese character is segmented separately). To obtain more accurate search results, use phrases with the syntax #"phrase to be searched for". After this function is disabled, all content is split based on delimiters. For example, assume that the log content is: error,400,I Know TodayIsMonday. After this function is disabled, the English content is split based on delimiters. The log is split into error, 400, I, Know, and TodayIsMonday. You can search for the log by error or TodayIsMonday. After this function is enabled, the background analyzer of LTS splits the log into error, 400, I, Know, Today, Is, and Monday. You can search for the log by error or Today.
   Delimiters	Splits the log content into multiple words based on the specified delimiters. If the default settings cannot meet your requirements, you can customize delimiters. All ASCII codes can be defined as delimiters. If you leave Delimiters blank, the field value is regarded as a whole. You can search for logs only through a complete string or by fuzzy match. Click Preview the see the effect. For example, assume that the log content is: error,400,I Know TodayIsMonday. If no delimiter is set, the entire log is regarded as a string error,400,I Know TodayIsMonday. You can search for the log only by the complete string error,400,I Know TodayIsMonday or by fuzzy match error,400,I K*. If the delimiter is set to a comma (,), the raw log is split into: error, 400, and I Know TodayIsMonday. You can search for the log by fuzzy or exact match, for example, error, 400, I Kn*, and TodayIs*. If the delimiters are set to a comma (,) and space, the raw log is split into: error, 400, I, Know, TodayIsMonday. You can search for the log by fuzzy or exact match, for example, Know, and TodayIs*.
   ASCII Delimiters	Click Add ASCII Delimiter and enter the ASCII value by referring to ASCII Table.

5. Click OK.

Configuring Index Fields

When creating a field index, you can add a maximum of 500 fields. A maximum of 100 subfields can be added for JSON fields.

Common and ASCII delimiters of field indexes are available only to whitelisted users. To use them, submit a service ticket.

1. After you configure full-text indexing, toggle on the switch under Log Analysis so that the configured field index supports visualized SQL analysis. Otherwise, visualized data after ICAgent structuring cannot be queried.
2. Specify the number of samples for quick analysis. The value ranges from 100,000 (default) to 10 million. Quick analysis provides a fast overview by sampling field value statistics rather than analyzing all log data. The more logs sampled, the slower the analysis.
3. Click Add Field under Index Fields to configure field indexing. For details about parameter settings, see Table 3.
   

   • The indexing parameters take effect only for the current field.
   • Index fields that do not exist in log content are invalid.
   • For details about system fields, see System Reserved Fields.
   • Automatically configuring field indexes: Click Auto Configure. LTS generates field indexes based on the first log event in the preview or common system reserved fields (such as hostIP, hostName, and pathFile). You can add or delete fields as required.
   • Configuring field indexes in batches: Select fields and click Batch Configure.

   Table 3 Custom index field parameters

   Parameter	Description
   Field	Log field name, such as level in the example log. The field name can contain only letters, digits, and underscores (_), and must start with a letter or underscore (_). The field name cannot contain double underscores (__). NOTE: Double underscores (__) are used in built-in reserved fields that are not displayed to users in LTS. Avoid using them in custom log field names, as this will prevent the configuration of field index names. LTS enables index fields for certain system reserved fields by default. For details, see System Reserved Fields. For system fields, system is displayed next to their names.
   Action	Displays the field status, including new, not modified, modified, and deleted. After the index field is changed, click Compare to view the differences between the original and modified configurations. New fields cannot be modified. When you modify the settings of Type, Case-Sensitive, Common Delimiters, ASCII Delimiters, Include Chinese, or Quick Analysis, the system compares the modified settings with the original settings and changes the action to modified. After you click OK, the fields whose action is deleted are not saved.
   Type	Data type of the log field value. The options are string, long, json, and float. NOTE: The JSON type takes effect only for ICAgent structuring parsing and does not take effect for cloud structuring parsing. Fields of long and float types do not support Case-Sensitivity, Include Chinese and Delimiters.
   Case-Sensitive	Indicates whether letters are case-sensitive during query. If this function is enabled, the query result is case-sensitive. For example, if the message field in the example log contains Know, you can query the log only with message:Know. If this function is disabled, the query result is case-insensitive. For example, if the message field in the example log contains Know, you can also query the log with message:KNOW or message:know.
   Common Delimiters	Splits the log content into multiple words based on the specified delimiters. If the default settings cannot meet your requirements, you can customize delimiters. All ASCII codes can be defined as delimiters. If you leave Delimiters blank, the field value is regarded as a whole. You can search for logs only through a complete string or by fuzzy match. For example, the content of the message field in the example log is I Know TodayIsMonday. If no delimiter is set, the entire log is regarded as a string I Know TodayIsMonday. You can search for the log only by the complete string message:I Know TodayIsMonday or by fuzzy search message:I Know TodayIs*. If the delimiter is set to a space, the raw log is split into: I, Know, and TodayIsMonday. You can find the log by fuzzy search or exact words, for example, message:Know, or message: TodayIsMonday.
   ASCII Delimiters	Click Add ASCII Delimiter and enter the ASCII value by referring to ASCII Table.
   Include Chinese	Indicates whether to distinguish between Chinese and English during query. After the function is enabled, if the log contains Chinese characters, the Chinese content is split based on unigram segmentation and the English content is split based on delimiters. NOTE: Unigram segmentation is to split a Chinese string into Chinese characters. The advantage of unigram segmentation is efficient word segmentation of massive logs, and other Chinese segmentation methods have great impact on the write speed. If Include Chinese is enabled, unigram segmentation is used for Chinese characters (each Chinese character is segmented separately). To obtain more accurate search results, use phrases with the syntax #"phrase to be searched for". After this function is disabled, all content is split based on delimiters. For example, the content of the message field in the example log is I Know TodayIsMonday. After this function is disabled, the English content is split based on delimiters. The log is split into I, Know, and TodayIsMonday. You can search for the log by message:Know or message:TodayIsMonday. After this function is enabled, the background analyzer of LTS splits the log into I, Know, Today, Is, and Monday. You can search for the log by message:Know or message:Today.
   Quick Analysis	By default, this option is enabled, indicating that this field will be sampled and collected. For details, see Creating an LTS Quick Analysis Task. NOTE: The principle of quick analysis is to collect statistics on 100,000 logs that match the search criteria, not all logs. The maximum length of a field for quick analysis is 2000 bytes. The quick analysis field area displays the first 100 records.
   Operation	Click to delete the target field.

   Figure 1 Batch configuration
   

4. Click OK.

System Reserved Fields

During log collection, LTS adds information such as the collection time, log type, and host IP address to logs in the form of Key-Value pairs. These fields are system reserved fields.

• When using APIs to write log data or add ICAgent configurations, avoid using the same field names as reserved field names to prevent issues such as duplicate field names and incorrect queries.
• LTS adds system reserved fields to log data at no cost. However, your usage of these fields will be billed on a pay-per-use basis. Enabling the index function for these fields will also generate a small amount of index traffic and storage fees. For more information, see Price Calculator.
• A custom log field must not contain double underscores (__) in its name. If it does, indexing cannot be configured for the field.

ASCII Table