Updated on 2024-03-08 GMT+08:00

Log Structuring

Log data can be structured or unstructured. Structured data is quantitative data or can be defined by unified data models. It has a fixed length and format. Unstructured data has no pre-defined data models and cannot be fit into two-dimensional tables of databases.

During log structuring, logs with fixed or similar formats are extracted from a log stream based on your defined structuring method and irrelevant logs are filtered out. You can then use SQL syntax to query and analyze the structured logs.

Prerequisites

Logs have been dumped to LTS by configuring Log Settings.

Procedure

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. In the navigation pane on the left, click and choose Security & Compliance > Cloud Firewall. The Dashboard page will be displayed.
  4. (Optional) If the current account has only one firewall instance, the firewall details page is displayed. If there are multiple firewall instances, click View in the Operation column to go to the details page.
  5. In the navigation pane on the left, choose Log Audit > Log Management. Select the target log group and log stream.
  6. Click the Visualization tab and select JSON.

    Figure 1 Log structuring

  7. Extract log fields.

    1. Click Step 1 Select a sample log event, select a log event, or enter a log event in the text box, and click OK.

      Select a typical log.

    2. Click Intelligent Extraction in Step 2 Extract fields to extract the log fields.
      Figure 2 Obtaining log fields
      • The float data type has seven digit precision.
      • To have higher accuracy, you are advised to change the field type to String when the accuracy exceeds seven digits.

  8. Click Save. The type of extracted fields cannot be changed after the structuring is complete.