Cloud Native Log Collection

Introduction

The Cloud Native Log Collection add-on (formerly log-agent) is developed based on Fluent Bit and OpenTelemetry for collecting logs and Kubernetes events. This add-on supports CRD-based log collection policies. It collects and forwards standard output logs, container file logs, node logs, and Kubernetes events in a cluster based on configured policies. It also reports Kubernetes events to AOM for configuring event alarms. By default, all abnormal events and some normal events are reported.

In versions 1.3.2 and later, Cloud Native Log Collection reports all warning events and some normal events to AOM by default. These events can be used to configure alarms.

If the cluster version is 1.19.16, 1.21.11, 1.23.9, 1.25.4, or later, after Cloud Native Log Collection is installed, the control plane components no longer report events to AOM. Instead, this add-on reports events to AOM. When uninstalling the add-on: If the add-on version is earlier than 1.7.3, the control plane will not resume event reporting, and events will no longer be reported to AOM. If the add-on version is 1.7.3 or later, the control plane will resume event reporting and continue to report events to AOM.

Log Collection Reliability

The log system's main purpose is to record all stages of data for service components, including startup, initialization, exit, runtime details, and exceptions. It is primarily employed in O&M scenarios for tasks like checking component status and analyzing fault causes.

Standard streams (stdout and stderr) and local log files use non-persistent storage. However, data integrity may be compromised due to the following risks:

• Log rotation and compression potentially deleting old files
• Temporary storage volumes being cleared when Kubernetes pods end
• Automatic OS cleanup triggered by limited node storage space

While the Cloud Native Log Collection add-on employs techniques like multi-level buffering, priority queues, and resumable uploads to enhance log collection reliability, logs could still be lost in the following situations:

• The service log throughput surpasses the collector's processing capacity.
• The service pod is terminated and immediately reclaimed by the container runtime.
• The log collector pod experiences exceptions.

The following lists some recommended best practices for cloud native log management. You can review and implement them thoughtfully.

• Use dedicated, high-reliable streams to record critical service data (for example, financial transactions) and store the data in persistent storage.
• Avoid storing sensitive information like customer details, payment credentials, and session tokens in logs.

Constraints

This add-on is available only in clusters v1.17 or later.

Add-on Performance

Permissions

The fluent-bit component of the add-on reads and collects the stdout logs on each node, file logs in pods, and node logs based on the collection configuration.

The following permissions are required for running the fluent-bit component:

• CAP_DAC_OVERRIDE: ignores the discretionary access control (DAC) restrictions on files.
• CAP_FOWNER: ignores the restrictions that the file owner ID must match the process user ID.
• DAC_READ_SEARCH: ignores the DAC restrictions on file reading and catalog search.
• SYS_PTRACE: allows all processes to be traced.

The Cloud Native Log Collection add-on enhances the log reading capability based on these permissions. However, if log files are stored in an external file system (such as SFS), these permissions may not take effect. To ensure that logs can be collected properly, ensure that the paas user (UID 10000) has the read and execute permissions on the directory where the logs are stored and the read permissions on the files in the directory.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. Locate Cloud Native Log Collection and click Install.
3. On the Install Add-on page, configure the specifications as needed.
   • If you select Preset, choose Small or Large based on the log volume on your node. CCE automatically configures the number of add-on pods and resource quotas according to the selected specification. You can view these configurations on the console.
   • If you selected Custom, you can adjust the number of pods and resource quotas as needed. High availability is not possible with a single pod. If an error occurs on the node where the add-on pod runs, the add-on will fail.

4. Configure deployment policies for the add-on pods.
   

   Scheduling policies do not take effect on the DaemonSet pods of the add-on.

   Table 1 Configurations for add-on scheduling

   Parameter

   Description

   Multi-AZ Deployment

   Preferred: Deployment pods of the add-on will be preferentially scheduled to nodes in different AZs. If all the nodes in the cluster are deployed in the same AZ, the pods will be scheduled to different nodes in that AZ. Forcible: Add-on pods are strictly distributed across AZs, with at most one pod per AZ. If the cluster's nodes do not span multiple AZs, some add-on pods will remain unscheduled and fail to run. If a node is faulty, the add-on pods on it may fail to be migrated.

5. Click Install.

Components

How to Use the Add-on

This add-on can collect container standard output logs, container file logs, node logs, and Kubernetes events. You can use LTS or AOM to store the collected logs. These services support different types of logs. For details, see Table 3.

Helpful Links

• After installing the Cloud Native Log Collection add-on, you can use the Logging to collect Kubernetes logs. For details, see Overview.
• During log collection, there may be issues like interrupted collection or data loss. To ensure log integrity and availability, you are advised to set log collection status monitoring and alarms. For details, see Configuring Metrics and Alarms for Log Collection Status.

Release History