Help Center/ Log Tank Service/ User Guide/ Log Ingestion/ Using ICAgent to Collect Logs/ Overview
Updated on 2024-09-19 GMT+08:00
View PDF
Share

Overview

You can use ICAgent to collect logs to LTS. When creating a log ingestion task, you can customize collection policies, such as parsing, whitelist, and blacklist rules, and raw log uploading settings. An ICAgent collection configuration defines the process of collecting logs of the same type from a server, parsing them, and sending them to a specified log stream.

ICAgent Collection Principles

Once ICAgent is installed, LTS listens to the corresponding log files in real time. After receiving a collection configuration delivered by the LTS console, ICAgent periodically parses the configuration, obtains the collection path, matches the path on the node, monitors the corresponding files of the node, and detects the files' changes through polling and inotify mechanisms.

Upon detecting a file change, ICAgent reads the file content, divides the content into blocks, and sends the blocks to the processing module for log processing, such as single-line, multi-line, structuring, splitting, and tagging. Then, ICAgent submits the processed task to the sending task pool to report the task to LTS.

ICAgent Installation

To collect host metrics, container metrics, node logs, container logs, and standard output logs from hosts, you need to install ICAgent on the hosts.

For details, see Installing ICAgent (Intra-Region Hosts) and Installing ICAgent (Extra-Region Hosts).

  • An intra-region host is in the same region as the LTS console, for example, both in CN-Hong Kong.
  • An extra-region host is not in the same region as the LTS console, for example, a host in another Huawei Cloud region or provided by a third party.

Advantages

  • Collecting log data in non-intrusive mode as log files: You do not need to modify application code, and log collection does not affect application running.
  • Handling various exceptions during log collection: Security measures such as proactive retry and local cache are taken when a network or server exception occurs.
  • Centralized management: After installing ICAgent, you only need to set configurations such as host groups and ICAgent collection on the LTS console.
  • Comprehensive self-protection mechanisms: ICAgent is designed with strict limitations and protective measures regarding CPU, memory, and network usage, to minimize its impact on the performance of other services sharing the same server.

ICAgent Structuring Parsing Rules

Before configuring log ingestion, get to know the structuring parsing rules of ICAgent collection to facilitate your operations. LTS enables combined parsing, allowing you to create different structuring parsing rules for each collection configuration of a log stream. For details, see Setting ICAgent Structuring Parsing Rules.