Collecting Logs from CCE

LTS can collect logs from Cloud Container Engine (CCE).

Prerequisites

ICAgent has been installed in the CCE cluster and a host group with custom identifiers has been created for related nodes. If ICAgent has not been installed, upgrade it on the Host Management page. For details, see Upgrading ICAgent.
You have disabled Output to AOM.

Restrictions

Currently, ServiceStage hosting is not supported.
CCE cluster nodes whose container engine is Docker are supported. For details, see Cloud Container Engine (CCE)
CCE cluster nodes whose container engine is Containerd are supported. You must be using ICAgent 5.12.130 or later.
CCE Turbo clusters are supported. You must be using ICAgent 5.12.130 or later.
To collect container log directories mounted to host directories to LTS, you must configure the node file path.
Restrictions on the Docker storage driver: Currently, container file log collection supports only the overlay2 storage driver. devicemapper cannot be used as the storage driver. Run the following command to check the storage driver type:
```
docker info | grep "Storage Driver" 
```
If you select Fixed log stream for log ingestion, ensure that you have created a CCE cluster.

Procedure

Perform the following operations to configure CCE log ingestion:

Log in to the LTS console.
In the left navigation pane, choose Log Ingestion. On the displayed page, click CCE (Cloud Container Engine).
Alternatively, choose Log Management in the navigation pane. Click the name of the target log stream to go to the log details page. Click in the upper right corner. On the displayed page, click the Collection Configuration tab and click Create. In the displayed dialog box, click CCE (Cloud Container Engine).
Select a log stream.
Choose between Custom log stream and Fixed log stream to suite your requirements. Fixed log stream is recommended.

Custom log stream
1. Select a cluster from the CCE Cluster drop-down list.
2. Select a log group from the Log Group drop-down list. If there are no desired log groups, click Create Log Group to create one.
3. Select a log stream from the Log Stream drop-down list. If there are no desired log streams, click Create Log Stream to create one.
4. Click Next: Check Dependencies.
Fixed log stream

Logs will be collected to a fixed log stream. The default log streams of CCE clusters: stdout-{ClusterID} for standard output/errors, hostfile-{ClusterID} for node files, event-{ClusterID} for Kubernetes events, and containerfile-{ClusterID} for container files. Log streams are automatically named with a cluster ID. For example, if the cluster ID is Cluster01, the standard output/error log stream is stdout-Cluster01.

Log streams that can be created for a CCE cluster are stdout-{ClusterID} for standard output/errors, hostfile-{ClusterID} for node files, event-{ClusterID} for Kubernetes events, and containerfile-{ClusterID} for container files. If one of them has been created in a log group, the log stream will no longer be created in the same log group or other log groups.
1. Select a cluster from the CCE Cluster drop-down list.
2. The default log group is k8s-log-ClusterID. For example, if the cluster ID is c7f3f4a5-bcb8-11ed-a4ec-0255ac100b07, the default log group will be k8s-log-c7f3f4a5-bcb8-11ed-a4ec-0255ac100b07.
  
  If there is no such group, the system displays the following message: This log group does not exist and will be automatically created to start collecting logs.
3. Click Next: Check Dependencies.
  Figure 1 Fixed log stream
Check dependencies.
The system automatically checks whether the following items meet the requirements:
1. ICAgent has been installed (version 5.12.130 or later).
2. There is a host group with the same name and custom identifier k8s-log-ClusterID.
3. There is a log group named k8s-log-ClusterID.
4. There is a recommended log stream. If Fixed log stream is selected, this item is checked.
You need to meet all the requirements before moving on. If not, click Auto Correct.
- Auto Correct: Check the previous settings with one click.
- Check Again: Recheck dependencies.
- If Custom log stream is selected, the check item There is a log group named k8s-log-ClusterID is optional. Use the switch to enable or disable the check item.
(Optional) Select a host group.
1. Select one or more host groups from which you want to collect logs. If there are no desired host groups, click Create above the host group list to create one. For details, see Creating a Host Group (Custom Identifier).
  - The host group to which the cluster belongs is selected by default. You can also select host groups as required.
  - You can also deselect the host group. In this case, the collection configuration does not take effect. You are advised to select a host group during the first ingestion. You can skip this step and configure host groups after the ingestion configuration is complete. There are two options to do this:
    
    Choose Host Management in the navigation pane. Click the Host Groups tab and associate host groups with ingestion configurations.
    
    Choose Log Ingestion in the navigation pane and click an ingestion configuration. On the displayed page, add one or more host groups for association.
2. Click Next: Configurations.
Configure the collection.
Specify collection rules. For details, see Configuring the Collection.
(Optional) Configure log structuring.
For details, see Cloud Structuring Parsing.

If the selected log stream has been structured, exercise caution when deleting it.
(Optional) Configure indexes.
For details, see Index Settings.
Click Submit.

Configuring the Collection

When CCE is used to ingest logs, the configuration details are as follows:

Basic Information: Enter a name containing 1 to 64 characters. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed. The name cannot start with a period or underscore, or end with a period.

Data Source: Select a data source type and configure it.

Container standard output: Collects stderr and stdout logs of a specified container in the cluster.
- The standard output of the matched container is collected to the specified log stream. Standard output to AOM stops.
- The container standard output must be unique to a host.
Container file: Collects file logs of a specified container in the cluster.
Node file: Collects files of a specified node in the cluster.

You cannot add the same host path to more than one log stream.
Kubernetes event: Collects event logs in the Kubernetes cluster.

Kubernetes events of a Kubernetes cluster can be ingested to only one log stream.

**Table 1** Collection configuration parameters
Parameter	Description
Container standard output	Collects container standard output to AOM, and collects stderr and stdout logs of a specified container in the cluster. Collecting container standard output to AOM: ICAgent is installed on hosts in the cluster by default, and logs is collected to AOM. The function of collecting container standard output to AOM is enabled. Disable this function to collect stdout streams to LTS. Either Container Standard Output (stdout) or Container Standard Error (stderr) must be enabled.
Container file	Collection Paths: LTS collects logs from the specified paths. NOTE: If a container mount path has been configured for the CCE cluster workload, the paths added for this field are invalid. The collection paths take effect only after the mount path is deleted. You cannot add the same host path to more than one log stream. Add custom wrap rule: ICAgent determines whether a file is wrapped based on the file name rule. If your wrapping rule does not comply with the built-in rules, you can add a custom wrap rule to prevent log loss during repeated collection and wrapping. The built-in rules are {basename}{connector}{wrapping identifier}.{suffix} and {basename}.{suffix}{connector}{wrapping identifier}. The connector is -._, the wrapping identifier is a non-letter symbol, and the suffix is a letter. A custom wrapping rule consists of {basename} and the feature regular expression of the wrapped file. The following example and screenshot are for reference only. Example: If your log file name is /opt/test.out.log, and the wrapped file names are test.2024-01-01.0.out.log and test.2024-01-01.1.out.log, the collection path is */opt/.log** and the wrapping rule is {basename}\.[-0-9\.].out.log. Set Collection Filters: Blacklisted directories or files will not be collected. If you specify a directory, all files in the directory are filtered out.
Node file	Collection Paths: LTS collects logs from the specified paths. NOTE: You cannot add the same host path to more than one log stream. Add custom wrap rule: ICAgent determines whether a file is wrapped based on the file name rule. If your wrapping rule does not comply with the built-in rules, you can add a custom wrap rule to prevent log loss during repeated collection and wrapping. The built-in rules are {basename}{connector}{wrapping identifier}.{suffix} and {basename}.{suffix}{connector}{wrapping identifier}. The connector is -._, the wrapping identifier is a non-letter symbol, and the suffix is a letter. A custom wrapping rule consists of {basename} and the feature regular expression of the wrapped file. The following example and screenshot are for reference only. Example: If your log file name is /opt/test.out.log, and the wrapped file names are test.2024-01-01.0.out.log and test.2024-01-01.1.out.log, the collection path is */opt/.log** and the wrapping rule is {basename}\.[-0-9\.].out.log. Set Collection Filters: Blacklisted directories or files will not be collected. If you specify a directory, all files in the directory are filtered out.
Kubernetes event	You do not need to configure this parameter. Only ICAgent 5.12.130 or later is supported.

Kubernetes Matching Rules: Set these parameters only when the data source type is set to Container standard output or Container file.

After entering a regular expression matching rule, click the button of verification to verify the regular expression.

**Table 2** Kubernetes matching rules
Parameter	Description
Namespace Name Regular Expression	Specifies the container whose logs are to be collected based on the namespace name. Regular expression matching is supported. NOTE: LTS will collect logs of the namespaces with names matching this expression. To collect logs of all namespaces, leave this field empty.
Pod Name Regular Expression	Specifies the container whose logs are to be collected based on the pod name. Regular expression matching is supported. NOTE: LTS will collect logs of the pods with names matching this expression. To collect logs of all pods, leave this field empty.
Container Name Regular Expression	Specifies the container whose logs are to be collected based on the container name (the Kubernetes container name is defined in spec.containers). Regular expression matching is supported. NOTE: LTS will collect logs of the containers with names matching this expression. To collect logs of all containers, leave this field empty.
Label Whitelist	Specifies the containers whose logs are to be collected. If you want to set a Kubernetes label whitelist, Label Key is mandatory and Label Value is optional. NOTE: LTS will match all containers with a Kubernetes label containing a specified Label Key with an empty corresponding Label Value. If Label Value is not empty, only containers with a Kubernetes label containing a specified Label Key that is equal to its Label Value are matched with LTS. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple whitelists is based on an OR operation, meaning that a Kubernetes label can be matched as long as it meets any of the whitelists.
Label Blacklist	Specifies the containers whose logs are not to be collected. If you want to set a Kubernetes label blacklist, Label Key is mandatory and Label Value is optional. NOTE: LTS will exclude all containers with a Kubernetes label containing a specified Label Key with an empty corresponding Label Value. If Label Value is not empty, only containers with a Kubernetes label containing a specified Label Key that is equal to its Label Value will be excluded. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple blacklists is based on an OR operation, meaning that a Kubernetes label can be excluded as long as it meets any of the blacklists.
Kubernetes Label	After the Kubernetes Label is set, LTS adds related fields to logs. NOTE: LTS adds the specified fields to the log when each Label Key has a corresponding Label Value. For example, if you enter "app" as the key and "app_alias" as the value, when the container label contains "app=lts", "{app_alias: lts}" will be added to the log.
Container Label Whitelist	Specifies the containers whose logs are to be collected. If you want to set a container label whitelist, Label Key is mandatory and Label Value is optional. NOTE: LTS will match all containers with a container label containing a specified Label Key with an empty corresponding Label Value. If Label Value is not empty, only containers with a container label containing a specified Label Key that is equal to its Label Value are matched with LTS. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple whitelists is based on an OR operation, meaning that a container label can be matched as long as it meets any of the whitelists.
Container Label Blacklist	Specifies the containers whose logs are not to be collected. If you want to set a container label blacklist, Label Key is mandatory and Label Value is optional. NOTE: LTS will exclude all containers with a container label containing a specified Label Key with an empty corresponding Label Value. If Label Value is not empty, only containers with a container label containing a specified Label Key that is equal to its Label Value will be excluded. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple blacklists is based on an OR operation, meaning that a container label can be excluded as long as it meets any of the blacklists.
Container Label	After the Container Label is set, LTS adds related fields to logs. NOTE: LTS adds the specified fields to the log when each Label Key has a corresponding Label Value. For example, if you enter "app" as the key and "app_alias" as the value, when the container label contains "app=lts", "{app_alias: lts}" will be added to the log.
Environment Variable Whitelist	Specifies the containers whose logs are to be collected. If you want to set an environment variable whitelist, Label Key is mandatory and Label Value is optional. NOTE: LTS will match all containers with environment variables containing either an Environment Variable Key with an empty corresponding Environment Variable Value, or an Environment Variable Key with its corresponding Environment Variable Value. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple whitelists is based on an OR operation, meaning that a container environment variable can be matched as long as it meets any of key-value pairs.
Environment Variable Blacklist	Specifies the containers whose logs are not to be collected. If you want to set an environment variable blacklist, Label Key is mandatory and Label Value is optional. NOTE: LTS will exclude all containers with environment variables containing either an Environment Variable Key with an empty corresponding Environment Variable Value, or an Environment Variable Key with its corresponding Environment Variable Value. Label Key requires full matching while Label Value supports regular matching. The relationship between multiple blacklists is based on an OR operation, meaning that a container environment variable can be excluded as long as it meets any of key-value pairs.
Environment Variable Label	After the environment variable label is set, the log service adds related fields to the log. NOTE: LTS adds the specified fields to the log when each Environment Variable Key has a corresponding Environment Variable Value. For example, if you enter "app" as the key and "app_alias" as the value, when the Kubernetes environment variable contains "app=lts", "{app_alias: lts}" will be added to the log.

Perform other configurations.

**Table 3** Other configurations
Parameter	Description
Max Directory Depth	The maximum directory depth is 5 levels. The LogListener does not collect log files with directory levels beyond this value. Set to the appropriate level for a target collection path with fuzzy matching strings to avoid wasted resources.
Split Logs	LTS supports log splitting, which is disabled by default. If this option is enabled, a single-line log larger than 500 KB will be split into multiple lines for collection. For example, a line of 600 KB log will be split into two lines for collection, the first line 500 KB and the second line 100 KB. If this option is disabled, a log larger than 500 KB will be truncated.
Collect Binary Files	LTS supports binary file collection, which is disabled by default. Run the file -i File_name command to view the file type. charset=binary indicates that a log file is a binary file. If this option is enabled, binary log files will be collected, but only UTF-8 strings are supported. Other strings will be garbled on the LTS console. If this option is disabled, binary log files will not be collected.
Log File Code	The encoding format of log files is UTF-8.
Collection Policy	Select incremental or full collection. Incremental: When collecting a new file, ICAgent reads the file from the end of the file. Full: When collecting a new file, ICAgent reads the file from the beginning of the file.

Configure the log format and log time.

**Table 4** Log collection settings
Parameter	Description
Log Format	Single-line: Each log line is displayed as a single log event. Multi-line: Multiple lines of exception log events can be displayed as a single log event. This is helpful when you check logs to locate problems.
Log Time	System time: log collection time by default. It is displayed at the beginning of each log event. NOTE: Log collection time is the time when logs are collected and sent by ICAgent to LTS. Log printing time is the time when logs are printed. ICAgent collects and sends logs to LTS with an interval of 1 second. Restriction on log collection time: Logs are collected within 24 hours before and after the system time.
Log Time	Time wildcard: You can set a time wildcard so that ICAgent will look for the log printing time as the beginning of a log event. If the time format in a log event is 2019-01-01 23:59:59.011, the time wildcard should be set to YYYY-MM-DD hh:mm:ss.SSS. If the time format in a log event is 19-1-1 23:59:59.011, the time wildcard should be set to YY-M-D hh:mm:ss.SSS. NOTE: If a log event does not contain year information, ICAgent regards it as printed in the current year. Example: YY - year (19) YYYY - year (2019) M - month (1) MM - month (01) D - day (1) DD - day (01) hh - hours (23) mm - minutes (59) ss - seconds (59) SSS - millisecond (999) hpm - hours (03PM) h:mmpm - hours:minutes (03:04PM) h:mm:sspm - hours:minutes:seconds (03:04:05PM) hh:mm:ss ZZZZ (16:05:06 +0100) hh:mm:ss ZZZ (16:05:06 CET) hh:mm:ss ZZ (16:05:06 +01:00)
Log Segmentation	This parameter needs to be specified if the Log Format is set to Multi-line. By generation time indicates that a time wildcard is used to detect log boundaries, whereas By regular expression indicates that a regular expression is used.
Regular Expression	You can set a regular expression to look for a specific pattern to indicate the beginning of a log event. This parameter needs to be specified when you select Multi-line for Log Format and By regular expression for Log Segmentation.

The time wildcard and regular expression will look for the specified pattern right from the beginning of each log line. If no match is found, the system time, which may be different from the time in the log event, is used. In general cases, you are advised to select Single-line for Log Format and System time for Log Time.

Parent topic: Collecting Logs from Cloud Services

Previous topic: Collecting Logs from CBH

Next topic: Collecting Logs from CDN