Generating Metrics Based on Logs

Background

• You can create metric rules on the LTS console to generate statistical reports as required. You can also set a single log filter or add associations and groups to set multiple log filters to retain logs that meet the filters. Statistics on the structured logs within a specified time range can be collected dynamically and displayed on the Prometheus instances of AOM, which is easy to operate and powerful.
• Each metric rule can generate only one result. To obtain multiple results, you need to create multiple metric rules.

Prerequisites

• A Prometheus instance has been created on the AOM console.
• Logs have been ingested to LTS.
• Log data has been structured. Currently, only structured log data can be used to generate metrics.
  

  To generate metrics based on logs, the log time sequence deviation must be within a small range (5s if Frequency is set to 5s, 1 minute if Frequency is set to 1 min, and 1 minute and 30 seconds if Frequency is set to 5 min). It is recommended that ICAgent be used to report logs in structured mode. If cloud structuring parsing is used, the log time may be out of order. As a result, logs cannot be processed within the statistical period and the statistical result is inaccurate.

Restrictions

A single user can create a maximum of 10 log metric rules. The total number of metrics added to all rules cannot exceed 10.

Creating a Log Metric Rule

1. Log in to the LTS console.
2. Log in to the LTS console and choose Log Jobs in the navigation pane.
3. On the Metric Generation tab page, click Create Rule.
4. Configure basic log information.
   1. Enter a rule name. A rule name can contain a maximum of 256 characters, including letters, digits, underscores (_), and hyphens (-).
   2. Enter the description information.
   3. Retain the default setting of Status (enabled).
   4. Agency Authorization is enabled by default. To create a log metric generation task, authorize LTS and AOM to create cloud service agencies lts_admin_trust and aom_admin_trust.
   5. Enable Task Monitoring so that the execution status of each task is written to the log stream lts-system/lts-logtometric-statistics. You can view the monitoring center for metric generation tasks or configure alarm rules to detect exceptions that occur during processing in a timely manner.

5. Configure the data source.
   1. Select a source log group. If no group is available, click Create Log Group.
      

      • The logs that exceed the retention period will be automatically deleted. You can transfer logs to OBS buckets for long-term storage.
      • If your logs have not been ingested to LTS, create a log ingestion rule by referring to Log Ingestion and configure a log structuring parsing rule.
   2. Select a source log stream. If no stream is available, click Create Log Stream.
   3. Enable Log Sampling. This function randomly samples the log source at a rate ranging from 0.1 to 0.9.

6. Configure the metric storage location.
   1. Metrics generated from logs will be stored in AOM as user-defined metrics. Select the Prometheus instance where the metrics are to be stored. If no instance is available, click Create an instance.
   2. Customize a name for the log generation metric. The value can contain only letters, digits, underscores (_), and colons (:), and cannot start with a digit, underscore (_), or colon (:).
   3. Enter metric description.

7. Click Next.
8. Preview the information below Metric Preview. This simulates the metrics generated by executing SQL queries on log streams using the configured log filtering and statistics rules. To do so, you must first collect logs to LTS and configure structuring parsing rules and indexes. Otherwise, the preview is empty.
9. Implement statistical settings.
   1. Set a log filtering rule. Then you can preview the effect. If you cannot find the desired log fields in the Log Filtering and Log Statistics areas, configure them in the collection settings first by referring to Cloud Structuring Parsing.
      

      • You can filter logs using OR and AND conditions.
      • Different fields support different filtering rules.
      • The log filtering and statistics fields must be string, float, or long types, not json.
      • Log filtering is to reserve the logs that meet the conditions and discard the logs that do not meet the conditions.
      • The group by function must be string or integer types.
   2. Select or customize the field to count and specify Group by. The data collected one minute later than the allowed period will not be counted.
      

      • The following statistics types are supported:

        count: number of logs. countKeyword: number of times a keyword appears. sum: sum value of a specified field. avg: average value of a specified field. max: maximum value of a specified field. min: minimum value of a specified field. P50: the value below which 50% of the data falls. P75: the value below which 75% of the data falls. P90: the value below which 90% of the data falls. P95: the value below which 95% of the data falls. P99: the value below which 99% of the data falls.

      • For the P-series statistical type, the numbers are sorted and the value of xx% is used as the statistical result.

      • Log statistics are performed on logs that meet the conditions. During the execution, a log stream is generated in log group system. If the log stream is deleted, the execution details of all metric generation rules cannot be viewed.
   3. Set the frequency. Options: 5s, 1 min, or 5 min. The frequency not only represents the data reporting interval, but also represents the time window of the statistics operation. For example, 5 min means that after a 5-minute interval is reached, the data of the previous 5 minutes is collected and reported.
   4. After the setting is complete, the dimension is automatically generated based on the content selected in the query.
   5. Select a unit. For example, select a unit for angle, bandwidth, or frequency.

10. (Optional) Click Real-Time Log Preview to view real-time logs.
11. Click OK.
12. View the new rule on the Metric Generation tab page.
    

    • Metric visualization and alarm reporting: After a metric is created, you can go to the AOM console to configure a visualized metric chart on the Dashboard page or metric alarms on the Alarm Rules page.
    • You are advised to configure a maximum of five rules for each log stream.
    • You can click the buttons in the Operation column of the rule list to copy, modify, or delete a rule.