Updated on 2024-11-12 GMT+08:00

Collecting Kubernetes Events

The Cloud Native Logging add-on works with LTS to collect and store Kubernetes events and works with AOM to generate alarms.

Reporting Kubernetes Events to LTS

To enable Kubernetes event collection in different scenarios, take the following steps.

When enabling logging, you can select Kubernetes events to create a default log collection policy, so that all Kubernetes events are collected and reported to LTS.

  1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging.
  2. (Optional) If you are not authorized, obtain required permissions first.

    In the displayed dialog box, click Authorize.

    Figure 1 Authorize

  3. Click Enable and wait for about 30 seconds until the log page is automatically displayed.
    Figure 2 Enable
    • Stdout logs: A log collection policy named default-stdout will be created, which will report stdout logs from all namespaces to LTS.
    • Kubernetes events: A log collection policy named default-event will be created, which will report Kubernetes events from all namespaces to LTS.
    • To collect add-on logs (NGINX Ingress Controller stdout), you need to install NGINX Ingress Controller and enable logging for the add-on.

      After logging is enabled, a log collection policy named default-nginx-ingress will be created, which will report all nginx-ingress stdout logs with the collection label from all namespaces to LTS.

If logging has been enabled for a cluster but Kubernetes event collection has not been enabled, or the corresponding log collection policy has been deleted, you can manually create a log collection policy by taking the following steps:

  1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging.
  2. In the upper right corner, click View Log Collection Policies.

    All log collection policies are displayed.

  3. Click Create Log Collection Policy. Then, select Kubernetes events and click OK.
    Figure 3 Creating a log collection policy

  4. (After the creation is complete, you can view logs on the Logging page.) Select the log stream configured in the log collection policy to view the events reported to LTS.
    Figure 4 Viewing event details

Reporting Kubernetes Events to AOM

After Cloud Native Logging 1.3.2 is installed, all warning events and some normal events will be reported to AOM by default. The reported events can be used to configure alarms. If the cluster version is 1.19.16, 1.21.11, 1.23.9, 1.25.4, or later, after Cloud Native Logging is installed, events will be reported to AOM by this add-on instead of the control plane component. After Cloud Native Logging is uninstalled, events will not be reported to AOM.

Custom Event Reporting

If the reported events cannot meet requirements, you can modify the settings for the events.

  1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Settings.
  2. Click the Monitoring tab. In the Log Configuration area, modify the policy for reporting Kubernetes events to AOM.

    • Abnormal events: This option is enabled by default. All abnormal events are reported to AOM. You can click Configure Blocklist to add events that do not need to be reported to the blocklist. You can obtain event names in CCE Events.
    • Normal events: If this option is enabled, normal events will be reported to AOM. The system is pre-configured to report some normal events. If you need to customize the events to be reported, click Configure Trustlist to add the events to the trustlist. You can obtain event names in CCE Events.

  3. Click Confirm configuration.
  1. Run the following command on the cluster to modify the event collection settings:

    kubectl edit logconfig -n kube-system default-event-aom

  2. Modify the event collection settings as required.

    apiVersion: logging.openvessel.io/v1
    kind: LogConfig
    metadata:
      annotations:
        helm.sh/resource-policy: keep
      name: default-event-aom
      namespace: kube-system
    spec:
      inputDetail:    # Settings on CCE from which events are collected
        type: event    # Type of logs to be collected from CCE. Do not change the value.
        event:
          normalEvents:    # Used to configure normal events
            enable: true    # Whether to enable normal event collection
            includeNames:    # Names of events to be collected. If this parameter is not specified, all events will be collected.
            - NotTriggerScaleUp
            excludeNames:    # Names of events that are not collected. If this parameter is not specified, all events will be collected.
            - ScaleDown
          warningEvents:    # Used to configure warning events
            enable: true    # Whether to enable warning event collection
            includeNames:    # Names of events to be collected. If this parameter is not specified, all events will be collected.
            - NotTriggerScaleUp
            excludeNames:    # Names of events that are not collected. If this parameter is not specified, all events will be collected.
            - ScaleDown
      outputDetail:
        type: AOM    # Type of the system that receives the events. Do not change the value.
        AOM:
          events:
          - name: DeleteNodeWithNoServer    # Event name. This parameter is mandatory.
            resourceType: Namespace    # Type of the resource that operations are performed on.
            severity: Major    # Event severity after an event is reported to AOM, which can be Critical, Major, Minor, or Info. The default value is Major.