Ingesting Self-Built Kubernetes Application Logs to LTS

Prerequisites

• Ensure that the Helm v3 installation command has been executed in the Kubernetes cluster.
• Ensure that kubectl has been configured for the Kubernetes cluster.
• You have enabled ICAgent Diagnosis to view exceptions, overall status, and collection status of ICAgent. For details, see Setting ICAgent Collection.

Step 1: Select a Log Stream

1. Log in to the LTS console.
2. Choose Log Ingestion > Ingestion Center in the navigation pane. Then, click Self-built K8s - Application Logs.

   Alternatively, choose Log Ingestion > Ingestion Management in the navigation pane, and click Ingest Log > Self-built K8s - Application Logs.

   Alternatively, choose Log Management in the navigation pane and click the target log stream to access its details page. Click in the upper right corner. On the displayed page, click the Log Ingestion tab and click Ingest Log. In the displayed dialog box, click Self-built K8s - Application Logs.

3. Choose a collection mode between Fixed log stream (recommended) and Custom log stream.

   Table 1 Collection modes

   Collection Mode	Description
   Fixed log stream	Advantage: Logs of the same type are stored in the same stream for easy search. Disadvantages: Since logs of different structures are collected into the same stream, log structuring and SQL visual analysis are unavailable. Since each stream has a limited write rate of 100 MB/s, there may be performance bottlenecks in heavy traffic.
   Custom log stream	Advantages: Since logs of different structures are collected to different streams, log structuring and SQL visual analysis are available. Since the cumulative write rate of multiple streams is not limited, there are no performance bottlenecks in heavy traffic. Disadvantage: Complex management of a large number of log streams

   • If you set Collect to Fixed log stream, perform the following steps:

     Logs will be collected to a fixed log stream. The default log streams for a kubernetes cluster are stdout-{ClusterID} for standard output/errors, hostfile-{ClusterID} for node files, event-{ClusterID} for Kubernetes events, and containerfile-{ClusterID} for container files. Log streams are automatically named with a cluster ID. For example, if the cluster ID is Cluster01, the standard output/error log stream is stdout-Cluster01.

     Log streams that can be created for a kubernetes cluster are stdout-{ClusterID} for standard output/errors, hostfile-{ClusterID} for node files, event-{ClusterID} for Kubernetes events, and containerfile-{ClusterID} for container files. If one of them has been created in a log group, the log stream will no longer be created in the same log group or other log groups.

     1. Select Fixed log stream for Collect.
     2. Enter the cluster name and ID.
     3. Select a log group.
        

        If there is no such group, the system displays the following message: This log group does not exist and will be automatically created to start collecting logs.

     4. Click Next: Check Dependencies.
   • If you set Collect to Custom log stream, perform the following steps:
     1. Select Custom log stream.
     2. Enter the cluster name and ID.
     3. Select a log group from the Log Group drop-down list. If there are no desired log groups, click Create Log Group to create one.
     4. Select a log stream from the Log Stream drop-down list. If there are no desired log streams, click Create Log Stream to create one.
        Figure 1 Custom log stream
        
4. Click Next: Check Dependencies.

Step 2: Check Dependencies

1. The system automatically checks whether the following are met:
   • There is a host group with the custom identifier k8s-log-ClusterID.
   • There is a log group named k8s-log-ClusterID. The log retention period and description of the log group can be modified. If Fixed log stream is selected, this item is checked.
   • The recommended log stream exists. The log retention period and description of the log stream can be modified. If Fixed log stream is selected, this item is checked.
   You need to meet all the requirements before moving on. If not, click Auto Correct.

   

   • Auto Correct: a one-click option to finish the previous settings.
   • Check Again: Recheck dependencies.
   • If Custom log stream is selected, the check item There is a log group named k8s-log-ClusterID is optional. Toggle the switch to enable or disable the check.
2. Click Next: Install ICAgent.

Step 3: Install the Log Collection Component

On any host in the Kubernetes cluster, perform the following steps:

1. Obtain the ICAgent installation package.
   1. Obtain the ICAgent installation package (based on your actual information).

      wget https://icagent-{regionId}.{obsDomainName}/ICAgent_linux/icagentK8s-5.5.1.2.tar.gz

   2. Decompress the ICAgent installation package.

      tar -xzvf icagentK8s-5.5.1.2.tar.gz

   3. Go to the directory.

      cd icagentK8s

   4. Generate installation commands:

      Select the region of ingested logs.

      Select the project ID of the ingesting account.

      For Kubernetes Cluster, select Intra-Region.

2. Install ICAgent.
   1. Copy the ICAgent installation command. You must replace the AK/SK with the obtained one. Manually replace it when copying the command, or replace it as prompted when running the command.

      To prevent your AK/SK from being disclosed, select Turn off command history to prevent the AK/SK from being stored to disable historical record collection.

      Figure 2 Installing ICAgent
      
   2. Use a remote login tool (such as PuTTY) to log in to the target host as the root user and run the copied command.

      If the message ICAgent install success is displayed, the installation is successful. Then check the ICAgent status in the host list.

3. Click ICAgent Already Installed.

Step 4: (Optional) Select a Host Group

1. In the host group list, the host group to which the cluster belongs is selected by default. You can also select host groups as required.
2. Click Next: Configurations.

Step 5: Configure the Collection

When you configure log ingestion for self-built Kubernetes clusters, the collection configuration details are as follows.

1. Collection Configuration Name: Enter 1 to 64 characters. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed. Do not start with a period or underscore, or end with a period.
2. Data Source: Select a data source type and configure it.

   Table 2 Collection configuration parameters

   Type	Description
   Container standard output	Collects stderr and stdout logs of a specified container in the cluster. Collects stderr and stdout logs of a specified container in the cluster. Either Container Standard Output (stdout) or Container Standard Error (stderr) must be enabled. If you enable Container Standard Error (stderr), select your collection destination path: Collect standard output and standard error to different files (stdout.log and stderr.log) or Collect standard output and standard error to the same file (stdout.log). The standard output of the matched container is collected to the specified log stream. Standard output to AOM stops. Allow Repeated File Collection (not available to Windows) After you enable this function, one host log file can be collected to multiple log streams. This function is available only to certain ICAgent versions. For details, see Checking the ICAgent Version Description. After you disable this function, each collection path must be unique. That is, the same log file in the same host cannot be collected to different log streams.
   Container file	Collects file logs of a specified container path in a cluster. Collection Paths: Add one or more host paths. LTS collects logs from the specified paths. NOTE: If a container mount path has been configured for the CCE cluster workload, the paths added for this field are invalid. The collection paths take effect only after the mount path is deleted. Add Custom Wrapping Rule: ICAgent determines whether a file is wrapped based on the file name rule. If your wrapping rule does not comply with the built-in rules, you can add a custom wrap rule to prevent log loss during repeated collection and wrapping. The built-in rules are {basename}{connector}{wrapping identifier}.{suffix} and {basename}.{suffix}{connector}{wrapping identifier}. The connector is -._, the wrapping identifier is a non-letter symbol, and the suffix consists of letters. A custom wrapping rule consists of {basename} and the feature regular expression of the wrapped file. Example: If your log file name is /opt/test.out.log, and the wrapped file names are test.2024-01-01.0.out.log and test.2024-01-01.1.out.log, the collection path is /opt/*.log and the wrapping rule is {basename}\.[-0-9\.].out.log. Allow Repeated File Collection (not available to Windows) After you enable this function, one host log file can be collected to multiple log streams. This function is available only to certain ICAgent versions. For details, see Checking the ICAgent Version Description. After you disable this function, each collection path must be unique. That is, the same log file in the same host cannot be collected to different log streams. Set Collection Filters: Blacklisted directories or files will not be collected. If you specify a directory, all files in the directory are filtered out.
   Node file	Collects files of a specified node path in a cluster. Collection Paths: Add one or more host paths. LTS collects logs from the specified paths. Add Custom Wrapping Rule: ICAgent determines whether a file is wrapped based on the file name rule. If your wrapping rule does not comply with the built-in rules, you can add a custom wrap rule to prevent log loss during repeated collection and wrapping. The built-in rules are {basename}{connector}{wrapping identifier}.{suffix} and {basename}.{suffix}{connector}{wrapping identifier}. The connector is -._, the wrapping identifier is a non-letter symbol, and the suffix consists of letters. A custom wrapping rule consists of {basename} and the feature regular expression of the wrapped file. Example: If your log file name is /opt/test.out.log, and the wrapped file names are test.2024-01-01.0.out.log and test.2024-01-01.1.out.log, the collection path is /opt/*.log and the wrapping rule is {basename}\.[-0-9\.].out.log. Allow Repeated File Collection (not available to Windows) After you enable this function, one host log file can be collected to multiple log streams. This function is available only to certain ICAgent versions. For details, see Checking the ICAgent Version Description. After you disable this function, each collection path must be unique. That is, the same log file in the same host cannot be collected to different log streams. Set Collection Filters: Blacklisted directories or files will not be collected. If you specify a directory, all files in the directory are filtered out.
   Kubernetes event	Collects event logs of the Kubernetes cluster. You do not need to configure this parameter. Only ICAgent 5.12.150 or later is supported. NOTE: Kubernetes events of a Kubernetes cluster can be collected to only one log stream.

3. Set kubernetes matching rules only when the data source type is set to Container standard output or Container file.
   

   After entering a regular expression, click Verify to verify it.

   Table 3 Kubernetes matching rules

   Parameter	Description
   Namespace Name Regular Expression	Specifies the container whose logs are to be collected based on the namespace name. Regular expression matching is supported. NOTE: LTS will collect logs of the namespaces with names matching this expression. To collect logs of all namespaces, leave this field empty.
   Pod Name Regular Expression	Specifies the container whose logs are to be collected based on the pod name. Regular expression matching is supported. NOTE: LTS will collect logs of the pods with names matching this expression. To collect logs of all pods, leave this field empty.
   Container Name Regular Expression	Specifies the container whose logs are to be collected based on the container name (the Kubernetes container name is defined in spec.containers). Regular expression matching is supported. NOTE: LTS will collect logs of the containers with names matching this expression. To collect logs of all containers, leave this field empty.
   Label Whitelist	Specifies the containers whose logs are to be collected. If you want to set a Kubernetes label whitelist, Label Key is mandatory and Label Value is optional. When adding multiple whitelists, you can select the And or Or relationship. This means a container will be matched when it satisfies all or any of the whitelists. NOTE: If Label Value is empty, LTS will match all containers whose Kubernetes label contains a specified Label Key. If Label Value is not empty, only containers whose Kubernetes label contains a specified Label Key that is equal to its Label Value are matched. Label Key requires full matching while Label Value supports regular matching.
   Label Blacklist	Specifies the containers whose logs are not to be collected. If you want to set a Kubernetes label blacklist, Label Key is mandatory and Label Value is optional. When adding multiple blacklists, you can select the And or Or relationship. This means a container will be excluded when it satisfies all or any of the blacklists. NOTE: If Label Value is empty, LTS will exclude all containers whose Kubernetes label contains a specified Label Key. If Label Value is not empty, only containers whose Kubernetes label contains a specified Label Key that is equal to its Label Value will be excluded. Label Key requires full matching while Label Value supports regular matching.
   Kubernetes Label	After the Kubernetes Label is set, LTS adds related fields to logs. NOTE: LTS adds the specified fields to the log when each Label Key has a corresponding Label Value. For example, if you enter "app" as the key and "app_alias" as the value, when the container label contains "app=lts", "{app_alias: lts}" will be added to the log.
   Container Label Whitelist	Specifies the containers whose logs are to be collected. If you want to set a container label whitelist, Label Key is mandatory and Label Value is optional. When adding multiple whitelists, you can select the And or Or relationship. This means a container will be matched when it satisfies all or any of the whitelists. NOTE: If Label Value is empty, LTS will match all containers whose container label contains a specified Label Key. If Label Value is not empty, only containers whose container label contains a specified Label Key that is equal to its Label Value are matched. Label Key requires full matching while Label Value supports regular matching.
   Container Label Blacklist	Specifies the containers whose logs are not to be collected. If you want to set a container label blacklist, Label Key is mandatory and Label Value is optional. When adding multiple blacklists, you can select the And or Or relationship. This means a container will be excluded when it satisfies all or any of the blacklists. NOTE: If Label Value is empty, LTS will exclude all containers whose container label contains a specified Label Key. If Label Value is not empty, only containers whose container label contains a specified Label Key that is equal to its Label Value will be excluded. Label Key requires full matching while Label Value supports regular matching.
   Container Label	After the Container Label is set, LTS adds related fields to logs. NOTE: LTS adds the specified fields to the log when each Label Key has a corresponding Label Value. For example, if you enter "app" as the key and "app_alias" as the value, when the container label contains "app=lts", "{app_alias: lts}" will be added to the log.
   Environment Variable Whitelist	Specifies the containers whose logs are to be collected. If you want to set an environment variable whitelist, Label Key is mandatory and Label Value is optional. When adding multiple whitelists, you can select the And or Or relationship. This means a container will be matched when it satisfies all or any of the whitelists. NOTE: If Environment Variable Value is empty, LTS will match all containers whose environment variable contains a specified Environment Variable Key. If Environment Variable Value is not empty, only containers whose environment variable contains a specified Environment Variable Key that is equal to its Environment Variable Value are matched. Label Key requires full matching while Label Value supports regular matching.
   Environment Variable Blacklist	Specifies the containers whose logs are not to be collected. If you want to set an environment variable blacklist, Label Key is mandatory and Label Value is optional. When adding multiple blacklists, you can select the And or Or relationship. This means a container will be excluded when it satisfies all or any of the blacklists. NOTE: If Environment Variable Value is empty, LTS will exclude all containers whose environment variable contains a specified Environment Variable Key. If Environment Variable Value is not empty, only containers whose environment variable contains a specified Environment Variable Key that is equal to its Environment Variable Value will be excluded. Label Key requires full matching while Label Value supports regular matching.
   Environment Variable Label	After the environment variable label is set, the log service adds related fields to the log. NOTE: LTS adds the specified fields to the log when each Environment Variable Key has a corresponding Environment Variable Value. For example, if you enter "app" as the key and "app_alias" as the value, when the Kubernetes environment variable contains "app=lts", "{app_alias: lts}" will be added to the log.

4. Enable structuring parsing. For details, see Setting ICAgent Structuring Parsing Rules.

   LTS enables combined parsing, allowing you to create different structuring parsing rules for each collection configuration of a log stream.

   

   If you have configured cloud structuring parsing, delete its configurations before configuring ICAgent structuring parsing.

   Figure 3 ICAgent structuring parsing configuration
   
5. Set other configurations.

   Table 4 Other configurations

   Parameter	Description
   Max Directory Depth	The maximum directory depth is 20 levels. Collection paths can use double asterisks (**) for multi-layer fuzzy match. Specify the maximum directory depth in the text box. For example, if your log path is /var/logs/department/app/a.log and your collection path is /var/logs/**/a.log, logs will not be collected when this parameter is set to 1, but will be collected when this parameter is set to 2 or a larger number.
   Split Logs	LTS can split logs. You can set the value (size of a split log file) to up to 1,024 KB. In this example, the value is 500 KB. If this option is enabled, a single-line log larger than 500 KB will be split into multiple lines for collection. For example, a 600 KB single-line log will be split into a line of 500 KB and a line of 100 KB. If this option is disabled, when a log exceeds 500 KB, the extra part will be truncated and discarded.
   Collect Binary Files	LTS can collect binary files. Run the file -i File_name command to view the file type. charset=binary indicates that a log file is a binary file. If this option is enabled, binary log files will be collected, but only UTF-8 strings are supported. Other strings will be garbled on the LTS console. If this option is disabled, binary log files will not be collected.
   Log File Code	The log file encoding format can be UTF-8 or GBK (not available to Windows). UTF-8 encoding is a variable-length encoding mode and represents Unicode character sets. GBK, an acronym for Chinese Internal Code Extension Specification, is a Chinese character encoding standard that extends both the ASCII and GB2312 encoding systems.
   Collection Policy	Select Incremental or All. Incremental: When collecting a new file, ICAgent reads the file from the end of the file. Full: When collecting a new file, ICAgent reads the file from the beginning of the file.
   Custom Metadata	If this option is disabled, ICAgent will report logs to LTS based on the default system fields. You do not need to and cannot configure the fields. If this option is enabled, ICAgent will report logs based on your selected built-in fields and fields created with custom key-value pairs. Built-in Fields: Select built-in fields as required. Custom Key-Value Pairs: Click Add and set a key and value.

6. Configure the log format and time by referring to Table 5.

   Table 5 Log collection settings

   Parameter	Description
   Log Format	Single-line: Each log line is displayed as a single log event. Multi-line: Multiple lines of exception log events can be displayed as a single log event. This is helpful when you check logs to locate problems.
   Log Time	System time: log collection time by default. It is displayed at the beginning of each log event. NOTE: Log collection time is the time when logs are collected and sent by ICAgent to LTS. Log printing time is the time when logs are printed. ICAgent collects and sends logs to LTS with an interval of 1 second. Restriction on log collection time: Logs are collected within 24 hours before and after the system time.
   	Time wildcard: You can set a time wildcard so that ICAgent will look for the log printing time as the beginning of a log event. If the time format in a log event is 2019-01-01 23:59:59.011, the time wildcard should be set to YYYY-MM-DD hh:mm:ss.SSS. If the time format in a log event is 19-1-1 23:59:59.011, the time wildcard should be set to YY-M-D hh:mm:ss.SSS. NOTE: If a log event does not contain year information, ICAgent regards it as printed in the current year. Example: YY - year (19) YYYY - year (2019) M - month (1) MM - month (01) D - day (1) DD - day (01) hh - hours (23) mm - minutes (59) ss - seconds (59) SSS - millisecond (999) hpm - hours (03PM) h:mmpm - hours:minutes (03:04PM) h:mm:sspm - hours:minutes:seconds (03:04:05PM) hh:mm:ss ZZZZ (16:05:06 +0100) hh:mm:ss ZZZ (16:05:06 CET) hh:mm:ss ZZ (16:05:06 +01:00)
   Log Segmentation	This parameter needs to be specified if the Log Format is set to Multi-line. By generation time indicates that a time wildcard is used to detect log boundaries, whereas By regular expression indicates that a regular expression is used.
   Regular Expression	You can set a regular expression to look for a specific pattern to indicate the beginning of a log event. This parameter needs to be specified when you select Multi-line for Log Format and By regular expression for Log Segmentation.

   

   The time wildcard and regular expression will look for the specified pattern right from the beginning of each log line. If no match is found, the system time, which may be different from the time in the log event, is used. In general cases, you are advised to select Single-line for Log Format and System time for Log Time.

7. Click Next: Index Settings.

Step 6: Configure Indexing

1. (Optional) Configure indexing. For details, see Setting Indexes.
2. Click Submit.

Step 7: Complete the Ingestion Configuration

Setting Multiple Ingestion Configurations in a Batch

You can set multiple ingestion configurations for multiple scenarios in a batch, avoiding repetitive setups.

1. On the Ingestion Management page, click Batch Ingest to go to the details page. For details, see Table 6.

   Table 6 Adding configurations in batches

   Type	Parameter	Description
   Basic Settings	Ingestion Type	Select Self-built K8s - Application Logs.
   	Configurations to Add	Enter the number of ingestion configurations in the text box and click Add. A maximum of 100 ingestion configurations can be added, including the one already exists under Ingestion Settings by default. Therefore, you can add up to 99 more.
   Ingestion Settings	Configuration List	The ingestion configurations are displayed on the left. You can add up to 99 more configurations. The ingestion configuration details are displayed on the right. Set them by referring to Step 5: Configure the Collection. After an ingestion configuration is complete, you can click Apply to Other Configurations to copy its settings to other configurations.

2. Click Check Parameters. After the check is successful, click Submit.

   The added ingestion configurations will be displayed on the Ingestion Management page after the batch creation is successful.

3. (Optional) Perform the following operations on ingestion configurations:
   • Select multiple existing ingestion configurations and click Edit. On the displayed page, select an ingestion type to modify the corresponding ingestion configurations.
   • Select multiple existing ingestion configurations and click Enable or Disable. If you toggle off the switch in the Status column of an ingestion configuration, logs will not be collected for this configuration.
   • Select multiple existing ingestion configurations and click Delete.