Help Center/ Log Tank Service/ User Guide/ Log Search and Analysis/ Setting Cloud Structuring Parsing/ Overview
Updated on 2024-12-17 GMT+08:00
View PDF
Share

Overview

Log data can be structured or unstructured. Structured data is quantitative data or can be defined by unified data models. It has a fixed length and format. Unstructured data has no pre-defined data models and cannot be fit into two-dimensional tables of databases.

During log structuring, logs with fixed or similar formats are extracted from a log stream based on your defined structuring method and irrelevant logs are filtered out. You can then use SQL syntax to query and analyze the structured logs.

Log structuring parsing is a process of converting log data from unstructured or semi-structured to structured for better storage, query, and analysis, improving log data readability, searchability, and query efficiency.

Parsing Modes

LTS offers both cloud-based and ICAgent-based structuring parsing modes. However, a log stream can only be configured with one parsing mode. For example, if you have configured cloud structuring parsing for a log stream, delete the existing parsing configuration before configuring ICAgent structuring parsing. For details, see Figure 1.

If you have not configured structuring parsing when configuring log ingestion, you can configure ICAgent or cloud configuring parsing for the target log stream separately.

  • ICAgent structuring parsing is performed on the collection side and supports combined parsing of plug-ins. You can set multiple collection configurations with different structuring parsing rules for a single log stream. This parsing mode is recommended. For details, see Setting ICAgent Structuring Parsing Rules.
  • Leveraging the computing power of LTS servers, cloud structuring parsing structures logs in log streams using various log extraction methods. In the future, it will incur log processing traffic fees based on the log volume.
Figure 1 Different parsing modes

Precautions

  • Log structuring is performed on a per-log-stream basis.
  • Log structuring is recommended when most logs in a log stream share a similar pattern.
  • After the structuring configuration is modified, the modification takes effect only for newly written log data.