Overview

Applications and services generate extensive log data during running, including system statuses, errors, and user operation records. This data is crucial for system O&M, troubleshooting, and service analysis. However, the increasing volume and diverse formats of log data make management and analysis challenging. Ingesting logs to LTS helps efficiently collect and save this data. After logs are successfully ingested to LTS, you can use LTS's log search, analysis, and alarm functions to further enhance application O&M efficiency and gain business insights.

Video Tutorial

Billing

Log ingestion incurs fees. For details, see Billing Items.

Log Ingestion Modes

LTS enables real-time log ingestion via various modes. Logs can be collected using ICAgent, ingested from cloud services, or reported to LTS via SDKs or APIs. Subsequently, you can perform operations on these logs using the LTS console, such as searching and analyzing logs, visualizing log statistics in charts or dashboards, and setting alarm reporting and log transfer.

The following figure shows how data is collected by LTS.

Figure 1 Data collection
Click to enlarge

**Table 1** Data collection modes
Collection Mode	Description
Collecting logs via ICAgent	ICAgent is an intrusion-free log collection tool for LTS. It runs on hosts or clusters to collect host or containerized application logs to LTS for unified management. To collect logs from hosts, install ICAgent on the target hosts. For details, see Using ICAgent to Collect Host Logs. To collect logs from containers, install ICAgent in the target clusters. For details, see Using ICAgent to Collect Container Logs.
Ingesting cloud service logs to LTS	LTS collects log data from various cloud services, such as ELB, WAF, APIG, and CFW. The log data includes performance logs (resource usage) and network logs (traffic records). This enables unified management, quick search and analysis, and alarm rule configuration for improved O&M efficiency and insights into cloud service statuses. For details, see Ingesting Cloud Service Logs to LTS.
Importing data	You can import log files from OBS buckets to LTS either on a one-time or scheduled basis. For details, see Importing Log Files from OBS to LTS (Beta). NOTE: Currently, this function is in closed beta test and cannot be applied for.
Collecting logs via SDKs	LTS provides multi-language SDKs, such as Java, Go, Web, iOS, and Android SDKs, to improve development efficiency through encapsulation. SDKs across languages offer aligned functionalities, reducing multi-device adaptation costs. For details, see Using SDKs to Ingest Logs to LTS.
Collecting logs via APIs	You can call normal and high-precision log reporting APIs to report log data to LTS for central management. Log reporting API: reports logs generated at similar time points and in sequence. High-precision log reporting API: reports logs generated at different time points and not in sequence.
Ingesting logs across IAM accounts	You can create an agency to map log streams from a delegating account to a delegated account (your current LTS login).
Ingesting logs via the standard Kafka protocol	You can report logs to LTS by using the standard Kafka protocol. This method depends only on the Kafka protocol and works with various Kafka Producer SDKs and collection tools. For example, if you use Logstash for log collection, you only need to modify its configuration file to report logs to LTS. No additional components are required. For details, see Using Kafka to Report Logs to LTS.
Using anonymous write to collect logs	Use the anonymous write function to collect and analyze anonymous user information, such as browser details, browsing/purchase behaviors, and stay duration, from browsers and applets. For details, see Using Anonymous Write to Collect Logs.

Introduction to ICAgent

ICAgent is a log collection tool for LTS. It runs on hosts where logs need to be collected. ICAgent has the following advantages:

ICAgent collects existing log files without requiring modification to application code and affecting application running.
Handling various exceptions during log collection: Security measures such as proactive retry, local persistence of the collection cursor, and recording of the collection location are taken when a network or server exception occurs.
LTS-based centralized management: After installing ICAgent, you only need to set configurations such as host groups and ICAgent collection on the LTS console.
Comprehensive self-protection mechanisms: ICAgent is designed with strict limitations and protective measures regarding CPU, memory, and network usage, to minimize its impact on the performance of other services sharing the same server.

ICAgent collection principles

After receiving a collection configuration from the LTS ingestion page, ICAgent detects log file changes through polling and monitoring and listen for real-time updates to the associated files. ICAgent periodically parses the collection configuration, matches the specified collection paths to files on the local node, adds the files to a monitoring task, and detects the files' changes using both polling and inotify (a file system event monitoring mechanism provided by the Linux kernel).

Upon detecting a file change, ICAgent reads the file content, divides it into blocks, and sends them to the processing module. This module processes the content based on your collection configuration, performing tasks such as single-line, multi-line, structuring, splitting, and tagging. Then, ICAgent submits the processed data to the sending task pool for reporting to LTS.

Figure 2 ICAgent collection principles
Click to enlarge

ICAgent installation

To collect node logs, container logs, and standard output logs from hosts, you need to install ICAgent on the hosts.

For details, see Installing ICAgent (Intra-Region Hosts) and Installing ICAgent (Extra-Region Hosts).

An intra-region host is in the same region as the LTS console, for example, both in CN-Hong Kong.
An extra-region host is not in the same region as the LTS console, for example, a host in another Huawei Cloud region or provided by a third party.

ICAgent structuring parsing rules

In addition to log collection, ICAgent also structures log data during collection. Before log ingestion, understand the ICAgent structuring parsing rules and select one appropriate for your log content. For details, see Configuring ICAgent Structuring Parsing.

LTS also offers cloud structure parsing alongside ICAgent structure parsing. For details, see Setting Cloud Structuring Parsing. However, ICAgent structuring parsing is recommended for scenarios where ICAgent is used for log collection. Each log stream supports only one structuring method. For example, if you configure ICAgent structuring parsing for a log stream, you cannot enable cloud structuring parsing for the same stream.

ICAgent structuring parsing is performed on user hosts and supports combined plug-ins for parsing. You can set multiple collection configurations with different structuring parsing rules for a single log stream. This parsing mode is recommended. For details, see Configuring ICAgent Structuring Parsing.
Leveraging the computing power of LTS, cloud structuring parsing structures logs in log streams using various log extraction methods. In the future, it will incur log processing traffic fees based on the log volume.

For details about how to select structuring parsing modes for different log ingestion scenarios, see Figure 3.

Figure 3 Selecting structuring parsing modes
Click to enlarge