Help Center/ Log Tank Service/ User Guide/ Log Search and Analysis/ Cloud Structuring Parsing/ Overview
Updated on 2024-07-23 GMT+08:00
View PDF

Overview

Log data can be structured or unstructured. Structured data is quantitative data or can be defined by unified data models. It has a fixed length and format. Unstructured data has no pre-defined data models and cannot be fit into two-dimensional tables of databases.

During log structuring, logs with fixed or similar formats are extracted from a log stream based on your defined structuring method and irrelevant logs are filtered out. You can then use SQL syntax to query and analyze the structured logs.

Precautions

  • You have created a log stream.
  • Log structuring is recommended when most logs in a log stream share a similar pattern.
  • After the structuring configuration is modified, the modification takes effect only for newly written log data.

Creating a Structuring Rule

Add structuring rules to a log stream and LTS will extract logs based on the rules. You can then use the SQL syntax to query and analyze logs.

To structure logs:

  1. Log in to the LTS console and choose Log Management in the navigation pane on the left.
  2. Select a log group and a log stream.
  3. On the log stream details page, click in the upper right corner. On the page displayed, select Cloud Structuring Parsing to structure logs.

    You can then use SQL statements to query and analyze structured logs in the same way as you query and analyze data in two-dimensional database tables.

    • If Auto Configuration and Analysis is enabled, structured fields are used for field indexing and the quick analysis for fields is also enabled. The built-in fields hostIP, hostName, and pathFile are set as index fields by default.
    • If a structured field exceeds 20 KB, only the first 20 KB is retained.
    • The following system fields cannot be extracted during log structuring: groupName, logStream, lineNum, content, logContent, logContentSize, collectTime, category, clusterId, clusterName, containerName, hostIP, hostId, hostName, nameSpace, pathFile, and podName.

  4. Enable custom log time.
  5. Click Save.

Modifying a Structuring Rule

To modify a structuring rule, perform the following steps:

  1. On the Log Structuring page, click to modify a structuring rule.

    • You can modify the structuring rules, including the structuring mode, log extraction field, and tag field.
    • System templates cannot be modified.

  2. Click Save.

Deleting a Structuring Rule

If a log structuring rule is no longer used, perform the following steps to delete it:

  1. On the Log Structuring page, click to delete a structuring rule.
  2. In the displayed dialog box, click OK.

    Deleted structuring rules cannot be restored. Exercise caution when performing this operation.