Help Center/ Cloud Container Engine/ User Guide/ O&M/ O&M Best Practices/ Configuring Metrics and Alarms for Log Collection Status
Updated on 2025-09-05 GMT+08:00
View PDF

Configuring Metrics and Alarms for Log Collection Status

Log collection is usually configured for enterprise applications. However, it may be interrupted, and data may be lost. To ensure log integrity and availability, you are advised to configure metrics and alarms for log collection status (such as the number of error logs and the number of written logs). If there is an exception, an alarm notification is sent by email or SMS, so that you can quickly respond to and handle the problem.

Configuring Metrics for Log Collection Status

The Cloud Native Log Collection add-on uses the fluent-bit component to collect logs. You can configure metrics for the fluent-bit component to monitor the log collection status and detect problems in a timely manner.

  1. Enable the metric collection of fluent-bit.

    fluent-bit uses non-basic metrics. After metric collection is enabled, you will be billed for monitoring these metrics. For details, see AOM Pricing Details.

    1. Log in to the CCE console and click the cluster name to access the cluster console.
    2. In the navigation pane, choose Settings. On the Monitoring tab, enable Preset Policies.

    3. Click Manage under Preset Policies in Collection Settings.

    4. Enable preset policies for fluent-bit.

    5. Edit the trustlist to add the fluentbit_input_ingestion_paused metric. (If the Cloud Native Cluster Monitoring add-on version is 3.12.1 or later, skip this step.)

  2. Configure a dashboard on AOM or Grafana.

    You are advised to create a dashboard on AOM and add the node-level parameters listed in the following table for PromQL statements to collect statistics on log collection of each node for alarms.
    Table 1 Node-level parameters

    Parameter

    PromQL Statement

    Setting Note

    Bytes Written per Second

    sum(irate(fluentbit_input_bytes_total[2m])) by (pod)

    Small specification: < 5 MByte/s; large specification: < 10 MByte/s

    Logs Written per Second

    sum(irate(fluentbit_input_records_total[2m])) by (pod)

    Small specification: < 10,000/s; large specification: < 20,000/s

    Input Storage Limit Exceeded

    sum(fluentbit_input_storage_overlimit) by (pod)

    The value cannot be greater than 0 for a long time. Occasional non-zero values have no impacts.

    Input Paused

    sum(fluentbit_input_ingestion_paused) by (pod)

    The value cannot be greater than 0 for a long time. Occasional non-zero values have no impacts.

Configuring Alarms for Log Collection Status

You can configure alarms for log collection status on AOM so that you can respond to and handle problems in a timely manner.

Configuring Event Alarms

  1. Log in to the CCE console and click the cluster name to access the cluster console.
  2. In the navigation pane, choose Alarm Center. Then, choose Alarm Rules > Custom Alarm Rules, and click Create Alarm Rule.
  3. Set Rule Type to Event alarm. You are advised to configure the event alarms listed in the following table for log collection.

    Rule Name

    Event Name

    Troubleshooting

    Log collection policy deleted

    LogConfigDeleted

    If this alarm is generated, ensure that the deletion meets your expectation.

    Log collection policy execution error

    SanitizeLogConfigFailed

    If this alarm is generated, view the log-operator logs to locate the fault or contact technical support.
    Figure 1 Example for configuring an event alarm

  4. Set Triggering Mode to Immediate trigger and select the contact group to be notified. Configure other parameters as needed.
  5. Click OK.

Configuring Metric Alarms

Before configuring metric alarms, you need to configure metrics and a dashboard on AOM. For details, see Configuring Metrics for Log Collection Status.

  1. Log in to the CCE console and click the cluster name to access the cluster console.
  2. In the navigation pane, choose Alarm Center. Then, choose Alarm Rules > Custom Alarm Rules, and click Create Alarm Rule.
  3. Set Rule Type to Metric alarm. You are advised to configure the metric alarms listed in the following table for log collection.

    Rule Name

    PromQL Statement

    Troubleshooting

    Input storage limit exceeded

    sum(fluentbit_input_storage_overlimit) by (pod) > 0

    If this alarm is generated, check the node-level log collection status on the AOM dashboard.

    If the limit is reached, you are advised to reduce unnecessary log printing.

    If the limit is not reached, contact Huawei technical support.

    Input backpressure paused

    sum(fluentbit_input_ingestion_paused) by (pod) > 0

    If this alarm is generated, check the node-level log collection status on the AOM dashboard.

    If the limit is reached, you are advised to reduce unnecessary log printing.

    If the limit is not reached, contact Huawei technical support.
    Figure 2 Example for configuring a metric alarm

  4. Set Duration to 5 minutes and select the contact group to be notified. Configure other parameters as needed.
  5. Click OK.
Parent Topic: O&M Best Practices