How Do I Interconnect MRS with LTS?

Concepts

• ICAgent: ICAgent is a log collection agent provided by the LTS. It is installed on hosts to automatically collect log files from specified paths and send them to LTS. MRS cluster hosts must have ICAgent installed to enable log collection.
• Log group: A log group is the top-level container for log resources in LTS, used to classify and manage logs from different services or projects. Multiple log streams can be created under each log group.
• Log stream: A log stream is the smallest management unit for log data in LTS and belongs to a log group. Logs from the same component or of the same type are typically grouped into the same log stream for easier search and analysis.
• Host group: A host group is a logical grouping of multiple hosts in LTS, facilitating batch management of log collection configurations. By adding hosts from an MRS cluster to the same host group, log collection rules can be configured uniformly.

Prerequisites

You have obtained the account AK and SK. For details, see How Do I Obtain Access Keys (AK/SK)?

Interconnecting with LTS

1. Install ICAgent on the MRS node. For details, see Installing ICAgent (Cloud Hosts).
   

   For the first installation, install one server first, and then batch install all other hosts in the same way.

   After installation, run the following command on the host to confirm that the ICAgent process is running properly:

   ps -ef | grep icagent 

   If the command output contains the ICAgent process, the installation was successful.

2. Create a host group and add the ICAgent host installed in 1 to the host group.
   1. Log in to the LTS console.
   2. Click Host Management. On the displayed page, click Create Host Group in the upper right corner to create a host group.
   3. In the Create Host Group slide-out panel that is displayed, enter a host group name in the Host Group field and set Host OS to Linux or Windows.
   4. In the host list, select one or more hosts to add to the group and click OK.
      • You can filter hosts by host name or host IP address. You can also click Search by Host IP Address and enter multiple host IP addresses in the displayed search box to search for matches.
      • If your desired hosts are not in the list, click Install ICAgent. In the Install ICAgent slide-out panel displayed, install ICAgent on the hosts as prompted.
3. Create a log group.
   1. Log in to the LTS console.
   2. On the Log Management page, click Create Log Group.
   3. In the Create Log Group slide-out panel displayed, enter a log group name. A log group name:
      • Can contain only letters, numbers, hyphens (-), underscores (_), and periods (.).
      • Cannot start with a period (.) or underscore (_), and cannot end with a period (.).
      • Can contain 1 to 64 characters.
   4. Select an enterprise project. You can click View Enterprise Projects to view all enterprise projects.
   5. Set Log Retention Duration. The system stores logs for 30 days by default. You can change the log retention duration after the log group is created.

      LTS offers a free quota of 500 MB per month for log read/write, retention, and indexing. When the free quota is used up, you will be billed for excess usage on a pay-per-use basis.

   6. Click OK.
      • On the Log Management page, you can view log group details, including name, retention duration, creation time, creation type, and tags. You can also modify retention duration and tags. Click Modify in the Operation column of the log group to modify the name and retention duration.
      • Click the log group name to access the log stream details page.
4. Create a log stream.
   1. Log in to the LTS console.
   2. Click the drop-down button next to the log group name.
   3. Click Create Log Stream. In the Create Log Stream slide-out panel displayed, enter a log stream name. A log stream name:
      • Can contain only letters, numbers, hyphens (-), underscores (_), and periods (.).
      • Cannot start with a period (.) or underscore (_), and cannot end with a period (.).
      • Can contain 1 to 64 characters.
   4. Select an enterprise project. You can click View Enterprise Projects to view all enterprise projects.
   5. Click OK.

      On the log stream page, you can view details of the log stream, including log stream name, enterprise project, log retention duration, creation time, and creation type.

      

      You can configure different log streams for different components.

5. Add hosts for log ingestion.
   1. Log in to the LTS console.
   2. Choose Log Ingestion > Ingestion Center.
   3. Click Elastic Cloud Server (ECS).
   4. In the Select Log Stream step, set Log Group and Log Stream to the created log group and log stream name, and click Next.
   5. In the (Optional) Select Host Group step, select the host group created in 2 and click Next.
   6. In the Collections step, set Collection Configuration Name (used to identify this configuration in the collection configuration list) and Collection Paths, and click Next.
      

      Path configurations

      • You can configure multiple collection paths by clicking Add Collection Path.
      • Each collection path must be unique. That is, the same path of the same host cannot be configured for different log groups and log streams.
      • Collection paths support recursion. You can use double asterisks (**) to collect logs from up to five directory levels.
      • Collection paths support fuzzy match. You can use an asterisk (*) to represent one or more characters of a directory or file name.
      • If the collection path is set to a directory (such as /var/logs/), only .log, .trace, and .out files in the directory are collected.

      For example, configure the following collection paths:

      YARN task log path:

      • /srv/BigData/*/nm/containerlogs/**/container-localizer-syslog
      • /srv/BigData/*/nm/containerlogs/**/directory.info
      • /srv/BigData/*/nm/containerlogs/**/launch_container.sh
      • /srv/BigData/*/nm/containerlogs/**/prelaunch.err
      • /srv/BigData/*/nm/containerlogs/**/prelaunch.out
      • /srv/BigData/*/nm/containerlogs/**/stderr
      • /srv/BigData/*/nm/containerlogs/**/stdout
      • /srv/BigData/*/nm/containerlogs/**/syslog*
      • /srv/BigData/*/nm/containerlogs/**/*.log

      All HDFS service logs:

      • /var/log/Bigdata/audit/hdfs/jn/hdfs-audit-journalnode.log
      • /var/log/Bigdata/audit/hdfs/jn/SecurityAuth.audit
      • /var/log/Bigdata/hdfs/*/

      Configure log collection paths for other components by referring to HDFS service logs.

   7. Skip the Index Settings step and submit.
      

      For details about how to configure index settings, see the Log Tank Service User Guide.

      After completing the host access configuration, check the collection configuration status on the log ingestion page of the LTS management console to ensure it is normal. On the host management page, check whether the ICAgent status of the hosts in the host group is online.

6. View the logs.
   1. Log in to the LTS console.
   2. In the navigation pane, choose Log Management.
   3. In the Log Group Name column, click the name of the created log group to view logs.

      On the log viewing page, check whether the log records generated by the MRS cluster services are displayed properly. You can use the search function to search for specific keywords to verify the completeness of log collection.

Related Documents

• For complete usage guide for LTS, see Log Tank Service (LTS) User Guide.
• To configure alarm rules in LTS for log alarm notifications, see the alarm-related sections in the LTS documentation.
• For billing information about LTS, see Log Tank Service (LTS) Billing.