Help Center/ Log Tank Service/ Best Practices/ Log Alarms/ Using Multi-Chart Query Results for Alarm Detection and Notification
Updated on 2025-08-04 GMT+08:00

Using Multi-Chart Query Results for Alarm Detection and Notification

When configuring alarm rules, you may need to use the query results of multiple charts. You can use the query result of the first chart to detect alarms and include the query results of other charts in alarm notifications.

For example, you want to monitor error logs and trigger an alarm if 10 logs whose level is ERROR are generated within 5 minutes. The alarm notification email will then include the query result of error logs, including parameters level (log severity), num (log number), and message (error information).

To do this, you would add two charts to your alarm rule.

Figure 1 Chart details
  • First chart (numbered 0): queries the number of logs whose level is ERROR. The query result is used to define the detection rule's condition.
  • Second chart (numbered 1): queries error log details, including level, num, and message. These results will be included in the alarm notification email body.

You can follow the following steps to configure an alarm rule for this scenario.

Creating a Message Template

A message template provides a fixed format for your alarm notifications. When an alarm is triggered, the system automatically populates the template's variables with content from your alarm rules. For example, to include the query result of error logs in your email notifications, add the following statement to Body of your message template:

Query log: $event.annotations.results[1].raw_results
  1. Log in to the LTS console.
  2. Choose Log Alarms in the navigation pane.
  3. Click Alarm Notification Rules.
  4. Click the Message Templates tab and click Create. The Create Message Template right pane is displayed.
  5. Set parameters by referring to Table 1.

    Figure 2 Message template
    Table 1 Parameters for creating a message template

    Parameter

    Example Value

    Description

    Template Name

    multi_table_query

    Message template name.

    Use only digits, letters, underscores (_), and hyphens (-). Do not start or end with an underscore or hyphen. Enter up to 100 characters.

    Description

    -

    Description of a message template.

    Use only digits, letters, and underscores (_). Do not start or end with an underscore. Enter up to 1,024 characters.

    Message Header

    Chinese (Simplified)

    A header (for example, "Dear customer") will be added to each message by default.

    Notification method

    Email

    Notification method.

    Topic

    Huawei Cloud LTS Notification: [${region_name}] and ${starts_at} O&M Notification

    Email subject. You can specify a subject or use variables.

    A subject can contain a maximum of 512 characters.

    Variables:

    • Occurrence region: ${region_name}
    • Occurrence time: ${starts_at}

    Body

    Huawei Cloud account: ${domain_name};

    Alarm rule: <a href="$event.annotations.alarm_rule_url ">${event_name}</a>;

    Alarm status: $event.annotations.alarm_status;

    Alarm severity: ${event_severity};

    Occurrence time: ${starts_at};

    Expression: $event.annotations.condition_expression;

    Current value: $event.annotations.current_value;

    Statistical period: $event.annotations.frequency;

    Log group/stream name: $event.annotations.results[1].resource_id;

    Query statement: $event.annotations.results[1].sql;

    Query URL: <a href="$event.annotations.results[1].url ">Details</a>;

    Query log: $event.annotations.results[1].raw_results;

    You can insert variables into your email body.

    Variables:

    • Huawei Cloud account: ${domain_name}
    • Alarm rule details link: $event.annotations.alarm_rule_url
    • Original alarm rule name: ${event_name}
    • Alarm severity: ${event_severity}
    • Occurrence time: ${starts_at}
    • Alarm rule triggering condition expression: $event.annotations.condition_expression
    • Current value of the condition expression: $event.annotations.current_value
    • Statistical period: $event.annotations.frequency
    • To include the log group/stream name from the second chart (chart 1) for querying detailed error logs: $event.annotations.results[1].resource_id
    • To include the query URL from the second chart (chart 1) for querying detailed error logs, use this variable. Clicking the link will redirect you to the specific log stream details page: $event.annotations.results[1].url
    • To include the query log result from the second chart (chart 1) for querying detailed error logs: $event.annotations.results[1].raw_results

  6. Click OK.

Creating an Alarm Notification Rule

Alarm notification rules link SMN topics with message templates. When an alarm is triggered, the system automatically sends an email notification using the specified message template.

  1. Log in to the LTS console.
  2. Choose Log Alarms in the navigation pane.
  3. Click Alarm Notification Rules.
  4. On the Alarm Notification Rules tab page, click Create. Set parameters by referring to Table 2.

    Figure 3 Alarm notification rule
    Table 2 Alarm notification rule parameters

    Parameter

    Example Value

    Description

    Notification Rule

    error_log_email_notification

    Notification rule name, which cannot be changed after the rule is created. Naming rules:

    Enter 1 to 64 characters. Only digits, letters, underscores (_), and hyphens (-) are allowed. Do not start or end with an underscore or hyphen.

    Enterprise Project

    default

    Select an enterprise project.

    This parameter is displayed only when the enterprise project function is enabled for the current account.

    Description

    -

    Enter a description for the rule. Up to 1,024 characters are allowed.

    Topic

    LTS

    Select an SMN topic.

    You can click Create Topic to access the SMN console, create a topic named LTS, and add a subscription (recipient email addresses) for it.

    For details about how to create a topic, see Creating a Topic.

    For details about how to subscribe to a topic, see Adding a Subscription to a Topic.

    Message Templates

    multi_table_query

    Select a message template from the drop-down list.

  5. Click OK.

Creating an Alarm Rule

An alarm rule defines the conditions that trigger alarms, including the query condition, detection rule, statistical period, notification frequency, and notification channel.

  1. Log in to the LTS console.
  2. Choose Log Alarms in the navigation pane.
  3. Click the Alarm Rules tab.
  4. Click Create.
  5. On the displayed Create Alarm Rule right panel, set parameters as follows:

    1. Configure the Basic Info parameters.
      Figure 4 Basic Info
      Table 3 Basic information about search analysis alarms

      Parameter

      Example Value

      Description

      Rule Name

      error_log_alarm

      Define a name for your alarm rule based on service requirements. After the rule is created, move the cursor to the rule name in the rule list to view both the rule name and the original rule name. You can modify the rule name, but cannot modify the original rule name (defined during rule creation).

      Naming rules:

      Use only letters, digits, hyphens (-), and underscores (_). Do not start or end with a hyphen or underscore. Enter 1 to 128 characters.

      Enterprise Project Name

      default

      Select the required enterprise project. The default value is default.

      Description

      More than 10 ERROR-level logs were reported within 5 minutes. Log details are included.

      Brief description of the rule. Enter up to 128 characters.

    2. Configure the Statistical Analysis parameters.
      Figure 5 Statistical Analysis
      Table 4 Alarm rules based on Search | Analysis statistics

      Parameter

      Example Value

      Description

      Query Condition (query charts)

      Add the first chart (chart 0), which is used for alarm detection rules.

      lts-group-PR456

      Log Group: Select the created log group lts-group-PR456.

      lts-topic-PR456

      Log Stream: Select the created log stream lts-topic-PR456.

      5 minutes

      Query Time Range: Specify the query period of the statement. It is one period earlier than the current time. For example, if Query Time Range is set to 5 minutes and the current time is 9:00, the query statement period is 8:55–9:00.

      * | SELECT COUNT(*) AS error_count WHERE level = 'ERROR'

      Query Statement: Enter a statement in the following format.

      Search statement | SQL analysis statement

      LTS monitors logs in the log stream based on the configured statement. Enter a query statement and click Preview to see the query result.

      Click Add to add a chart (chart 1), which is used to display log content. The log group, log stream, and query time range should be the same as the previous chart. Set the query statement to:

      level:ERROR | SELECT level,num,message

      Check Rule

      $0.error_count > 10

      Enter a specific condition expression. When the expression execution result is true, an alarm is generated.

      When a condition expression is associated with multiple charts, the following format must be used to reference fields in the condition expression:

      ${Query condition No.}.{Query statement field}

      $0.error_count indicates the error_count field in the query statement * | SELECT COUNT(*) AS error_count WHERE level = 'ERROR' of the first chart (chart 0).

      The second chart (chart 1) is used to query the level, num, and message fields of error logs. It is not used for the alarm verification rule. The corresponding variables have been added to the body as instructed in Creating a Message Template.

      CAUTION:

      When multiple charts are used, the condition expression must contain the first chart (chart 0), that is, $0. Therefore, the sequence of the two charts in this example cannot be changed.

    3. Configure the Advanced Settings parameters.
      Figure 6 Advanced Settings
      Table 5 Parameters of a search analysis alarm rule

      Parameter

      Example Value

      Description

      Query Frequency

      Custom interval: 1 minute

      If you set the query frequency of a condition expression to Custom interval, you can specify the interval from 1 minute to 60 minutes or from 1 hour to 24 hours. For example, if the current time is 9:00 and the Custom interval is set to 5 minutes, the first query is at 9:00, the second query is at 9:05, the third query is at 9:10, and so on.

      Alarm Restored When

      3

      If alarm restoration notification is enabled and the trigger condition has not been met for the specified number of last queries, an alarm restoration notification is sent.

      Number of last queries: 1–10

      Notify When

      Select Alarm triggered and Alarm restored.

      • Alarm triggered: Specify whether to send a notification when an alarm is triggered. If this option is enabled, a notification will be sent when the trigger condition is met. If disabled, no notifications will be sent, even if the trigger condition is met.
      • Alarm restored: Specify whether to send a notification when an alarm is restored. If this option is enabled, a notification will be sent when the restoration policy is met. If disabled, no notifications will be sent, even if the restoration policy is met.

      Frequency

      Immediate

      Notification is sent as soon as an alarm is generated.

      Alarm Notification Rules

      error_log_email_notification

      Select the created alarm notification rule error_log_email_notification from the drop-down list.

      Language

      Chinese (Simplified)

      Specify the language (Chinese (simplified) or English) in which alarms are sent.

      Tag

      -

      Tag alarm rules as required. Click Add and enter a tag key and value.

  6. Click OK.

Viewing Alarm Notifications

Logs are reported to LTS. When 10 error logs are generated within 5 minutes, you will receive an email notification. The email body contains the error log statistics of the first chart (chart 0) and the error log details of the second chart (chart 1).