Overview

Applications and services generate extensive log data during running, including system statuses, errors, and user operation records. This data is crucial for system O&M, troubleshooting, and service analysis. However, the increasing volume and diverse formats of log data make management and analysis challenging. Ingesting logs to LTS helps efficiently collect and save this data. After log ingestion, you can use LTS's log search, analysis, and alarm functions to further enhance O&M efficiency and gain business insights.

Log Ingestion Modes

LTS enables real-time log ingestion via various modes. Logs can be collected using ICAgent, ingested from cloud services, or reported to LTS via APIs. Subsequently, you can perform operations on these logs using the LTS console, such as searching and analyzing logs, visualizing log statistics in charts or dashboards, and setting alarm reporting and log transfer.

**Table 1** Data collection modes
Collection Mode	Description
Collecting logs via ICAgent	ICAgent is an intrusion-free log collection tool for LTS. It runs on hosts or clusters to collect host or containerized application logs to LTS for unified management. To collect logs from hosts, install ICAgent on the target hosts. For details, see Using ICAgent to Collect Host Logs. To collect logs from containers, install ICAgent in the target clusters. For details, see Using ICAgent to Collect Container Logs.
Ingesting cloud service logs to LTS	LTS collects log data from various cloud services, such as AOM, APIG, and CFW. The log data includes operation logs (management event tracing), performance logs (resource usage), and network logs (traffic records). This enables unified management, quick search and analysis, and alarm rule configuration for improved O&M efficiency and insights into cloud service statuses. For details, see Ingesting Cloud Service Logs to LTS.
Collecting logs via APIs	You can call normal and high-precision log reporting APIs to report log data to LTS for central management. Log reporting API: reports logs generated at similar time points and in sequence. High-precision log reporting API: reports logs generated at different time points and not in sequence.
Ingesting logs across IAM accounts	You can create an agency to map log streams from a delegating account to a delegated account (your current LTS login). NOTE: This function is available only in regions AF-Johannesburg, AP-Singapore, CN-Hong Kong, CN East-Shanghai1, LA-Mexico City1, LA-Mexico City2, LA-Santiago, and LA-Sao Paulo1.
Ingesting logs via the standard Kafka protocol	This method relies only on the Kafka protocol and supports various Kafka Producer SDKs or collection tools. For example, if you are using Logstash to collect logs, simply modify its configuration to report logs to LTS. No additional components are required. For details, see Using Kafka to Report Logs to LTS.

Introduction to ICAgent

ICAgent is a log collection tool for LTS. It runs on hosts where logs need to be collected. ICAgent has the following advantages:

ICAgent collects existing log files without requiring modification to application code and affecting application running.
Handling various exceptions during log collection: Security measures such as proactive retry and local cache are taken when a network or server exception occurs.
Centralized management: After installing ICAgent, you only need to set configurations such as host groups and ICAgent collection on the LTS console.
Comprehensive self-protection mechanisms: ICAgent is designed with strict limitations and protective measures regarding CPU, memory, and network usage, to minimize its impact on the performance of other services sharing the same server.

ICAgent collection principles

Once being installed on a host, ICAgent listens to the corresponding log files in real time. After receiving a collection configuration delivered by the LTS console, ICAgent periodically parses the configuration, obtains the collection path, matches the path on the node, monitors the corresponding files of the node, and detects the file changes through polling and inotify mechanisms.

Upon detecting a file change, ICAgent reads the file content, divides the content into blocks, and sends the blocks to the processing module for log processing, such as single-line, multi-line, structuring, splitting, and tagging. Then, ICAgent submits the processed task to the sending task pool to report the task to LTS.

Figure 1 ICAgent collection principles
Click to enlarge

ICAgent installation

To collect host metrics, container metrics, node logs, container logs, and standard output logs from hosts, you need to install ICAgent on the hosts.

ICAgent structuring parsing rules

In addition to log collection, ICAgent also structures log data during collection. Before log ingestion, understand the ICAgent structuring parsing rules and select one appropriate for your log content. For details, see Configuring ICAgent Structuring Parsing.

LTS also offers cloud structure parsing alongside ICAgent structure parsing. For details, see Setting Cloud Structuring Parsing. However, ICAgent structuring parsing is recommended for scenarios where ICAgent is used for log collection. Each log stream supports only one structuring method. For example, if you configure ICAgent structuring parsing for a log stream, you cannot enable cloud structuring parsing for the same stream.

ICAgent structuring parsing is performed on the collection side and supports combined plug-ins for parsing. You can set multiple collection configurations with different structuring parsing rules for a single log stream. This parsing mode is recommended. For details, see Configuring ICAgent Structuring Parsing.
Leveraging the computing power of LTS, cloud structuring parsing structures logs in log streams using various log extraction methods. In the future, it will incur log processing traffic fees based on the log volume.

Parent topic: Log Ingestion

Previous topic: Log Ingestion

Next topic: Using ICAgent to Collect Host Logs