Cloud Native Log Collection

Introduction

The Cloud Native Log Collection add-on (formerly log-agent) is developed based on Fluent Bit and OpenTelemetry for collecting logs and Kubernetes events. This add-on supports CRD-based log collection policies. It collects and forwards standard output logs, container file logs, node logs, and Kubernetes events in a cluster based on configured policies. It also reports Kubernetes events to AOM for configuring event alarms. By default, all abnormal events and some normal events are reported.

In 1.3.2 and later versions, Cloud Native Log Collection reports all warning events and some normal events to AOM by default. The reported events can be used to configure alarms. If the cluster version is v1.19.16, v1.21.11, v1.23.9, v1.25.4, or later, after Cloud Native Log Collection is installed, events are reported to AOM by this add-on instead of the control plane component. After this add-on is uninstalled, events will not be reported to AOM.

Log Collection Reliability

The log system's main purpose is to record all stages of data for service components, including startup, initialization, exit, runtime details, and exceptions. It is primarily employed in O&M scenarios for tasks like checking component status and analyzing fault causes.

Standard streams (stdout and stderr) and local log files use non-persistent storage. However, data integrity may be compromised due to the following risks:

• Log rotation and compression potentially deleting old files
• Temporary storage volumes being cleared when Kubernetes pods end
• Automatic OS cleanup triggered by limited node storage space

While the Cloud Native Log Collection add-on employs techniques like multi-level buffering, priority queues, and resumable uploads to enhance log collection reliability, logs could still be lost in the following situations:

• The service log throughput surpasses the collector's processing capacity.
• The service pod is terminated and immediately reclaimed by the container runtime.
• The log collector pod experiences exceptions.

The following lists some recommended best practices for cloud native log management. You can review and implement them thoughtfully.

• Use dedicated, high-reliable streams to record critical service data (for example, financial transactions) and store the data in persistent storage.
• Avoid storing sensitive information like customer details, payment credentials, and session tokens in logs.

Notes and Constraints

This add-on is available only in clusters v1.17 or later.

Add-on Performance

Permissions

The fluent-bit component of the add-on reads and collects the stdout logs on each node, file logs in pods, and node logs based on the collection configuration.

The following permissions are required for running the fluent-bit component:

• CAP_DAC_OVERRIDE: ignores the discretionary access control (DAC) restrictions on files.
• CAP_FOWNER: ignores the restrictions that the file owner ID must match the process user ID.
• DAC_READ_SEARCH: ignores the DAC restrictions on file reading and catalog search.
• SYS_PTRACE: allows all processes to be traced.

The Cloud Native Log Collection add-on enhances the log reading capability based on these permissions. However, if log files are stored in an external file system (such as SFS), these permissions may not take effect. To ensure that logs can be collected properly, ensure that the paas user (UID 10000) has the read and execute permissions on the directory where the logs are stored and the read permissions on the files in the directory.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. Locate Cloud Native Log Collection and click Install.
3. On the Install Add-on page, configure the specifications as needed.
   • If you select Preset, choose Small or Large based on your cluster scale. CCE automatically configures the number of add-on pods and resource quotas according to the selected specification. You can view these configurations on the console.

     You can select the Small option for clusters where the logs of a single node are less than 5000/s or 5 MB/s, and select the Large option for clusters where the logs on a single node are less than 10000/s or 10 MB/s.

   • If you selected Custom, you can adjust the number of pods and resource quotas as needed. High availability is not possible with a single pod. If an error occurs on the node where the add-on pod runs, the add-on will fail.

4. Configure Agency Settings. This configuration is supported by the add-on of 1.7.9 or later.
   Add-ons require runtime access to other cloud services. You need to configure an add-on agency to authorize these operations. For details, see Custom Agencies for Add-ons.

   • Create automatically: CCE automatically creates a default agency and uses this agency for the add-on. If there is a default agency, CCE will not create one again.
   • Use existing: Select one from the drop-down list. Ensure that the created agency has been assigned the required permissions. Otherwise, the add-on functions may be unavailable.

5. Configure add-on parameters, which are supported by the add-on of v1.6.1 or later. For details, see Tail.

   Parameter	Description	Default Value
   		Large Specification	Other Specification
   Initial Buffer Size (Buffer_Chunk_Size)	The size of the initial buffer used to read files.	256k	128k
   Maximum Buffer Size (Buffer_Max_Size)	Limit on the buffer size for each monitored file. When a buffer is added, the number of memory buffers that can be added is restricted by this value. If the limit is exceeded, the file will be removed from the monitored file list.	1024k	512k
   Memory Buffer Limit (Mem_Buf_Limit)	Memory limit when data is appended to the engine. When the limit is reached, the add-on will temporarily stop reading log file data. The reading will resume after the data is refreshed.	300mb	40mb

   

   The unit can be case-insensitive k, kb, m, mb, g or gb. If no unit is specified, the default unit of byte will be used.

6. Configure deployment policies for the add-on pods.
   

   Scheduling policies do not take effect on the DaemonSet pods of the add-on.

   Table 1 Configurations for add-on scheduling

   Parameter

   Description

   Multi-AZ Deployment

   Preferred: Deployment pods of the add-on will be preferentially scheduled to nodes in different AZs. If all the nodes in the cluster are deployed in the same AZ, the pods will be scheduled to different nodes in that AZ. Forcible: Add-on pods are strictly distributed across AZs, with at most one pod per AZ. If the cluster's nodes do not span multiple AZs, some add-on pods will remain unscheduled and fail to run. If a node is faulty, the add-on pods on it may fail to be migrated.

7. Click Install.

Components

How to Use the Add-on

This add-on can collect container standard output logs, container file logs, node logs, and Kubernetes events. You can use LTS or AOM to store the collected logs. These services support different types of logs. For details, see Table 3.

Helpful Links

• After installing the Cloud Native Log Collection add-on, you can use the Logging to collect Kubernetes logs. For details, see Overview.
• During log collection, there may be issues like interrupted collection or data loss. To ensure log integrity and availability, you are advised to set log collection status monitoring and alarms. For details, see Configuring Metrics and Alarms for Log Collection Status.

Release History