How Do I Interconnect MRS with LTS?

Prerequisites

You have obtained an AK/SK pair by referring to How Do I Obtain an AK/SK Pair?

Procedure

1. Install ICAgent on the MRS node. For details, see Installing ICAgent (HUAWEI CLOUD Hosts).
   

   For the first installation, install one server first, and then batch install all other hosts in the same way.

2. Create a host group and add the installed ICAgent host in 1 to the host group.
   1. Log in to the Log Tank Service (LTS) console, and choose Host Management in the left navigation pane. On the displayed page, click Create Host Group in the upper right corner.
   2. In the Create Host Group slide-out panel that is displayed, enter a host group name in the Host Group field and set Host OS to Windows.
   3. In the host list, select one or more hosts to add to the group and click OK.
      • You can filter hosts by host name or host IP address. You can also click Search by Host IP Address and enter multiple host IP addresses in the displayed search box to search for matches.
      • If your desired hosts are not in the list, click Install ICAgent. In the Install ICAgent slide-out panel displayed, install ICAgent on the hosts as prompted.
3. Create a log group.
   1. Log in to the LTS console, choose Log Management in the navigation pane on the left, and click Create Log Group in the upper right corner.
   2. In the Create Log Group slide-out panel displayed, enter a log group name. A log group name:
      • Can contain only letters, numbers, hyphens (-), underscores (_), and periods (.).
      • Cannot start with a period (.) or underscore (_), and cannot end with a period (.).
      • Can contain 1 to 64 characters.
   3. Set Log Retention Duration to 1 to 30 days. If this parameter is not specified, logs are retained for 7 days by default. You can change the log retention duration after the log group is created.

      LTS offers a free quota of 500 MB per month for log read/write, retention, and indexing. When the free quota is used up, you will be billed for excess usage on a pay-per-use basis.

   4. Click OK.
      • In the log group list on the Log Management page, you can view details of the log group, including log group name, log retention duration, creation time, creation type, and tags. Log retention duration and tags can be modified. Click Modify in the Operation column of the log group to modify the duration.
      • Click the log group name to access the log stream details page.
4. Create a log stream.
   1. On the LTS console, click the expansion button of a log group.
   2. Click Create Log Stream. In the Create Log Stream slide-out panel displayed, enter a log stream name. A log stream name:
      • Can contain only letters, numbers, hyphens (-), underscores (_), and periods (.).
      • Cannot start with a period (.) or underscore (_), and cannot end with a period (.).
      • Can contain 1 to 64 characters.
   3. Select an enterprise project. You can click View Enterprise Projects to view all enterprise projects.
   4. Click OK.

      On the log stream page, you can view details of the log stream, including log stream name, enterprise project, log retention duration, creation time, and creation type.

      

      You can configure different log streams for different components.

5. Add hosts for log ingestion.
   1. In the left navigation pane of the LTS console, choose Log Ingestion.
   2. On the All tab page displayed, click Elastic Cloud Server (ECS).
   3. Add the following configuration to Collection Path:

      YARN task log path:

      /srv/BigData/*/nm/containerlogs/**/container-localizer-syslog

      /srv/BigData/*/nm/containerlogs/**/directory.info

      /srv/BigData/*/nm/containerlogs/**/launch_container.sh

      /srv/BigData/*/nm/containerlogs/**/prelaunch.err

      /srv/BigData/*/nm/containerlogs/**/prelaunch.out

      /srv/BigData/*/nm/containerlogs/**/stderr

      /srv/BigData/*/nm/containerlogs/**/stdout

      /srv/BigData/*/nm/containerlogs/**/syslog*

      /srv/BigData/*/nm/containerlogs/**/*.log

      

      Path configuration:

      • Logs can be collected recursively. A double asterisk (**) can represent up to 5 directory levels in a path.
      • You can use an asterisk (*) as a wildcard for fuzzy match. The wildcard (*) can represent one or more characters of a directory or file name.
      • If the collection path is set to a directory (such as /var/logs/), only .log, .trace, and .out files in the directory are collected.

      Component logs:

      All HDFS service logs:

      /var/log/Bigdata/audit/hdfs/jn/hdfs-audit-journalnode.log

      /var/log/Bigdata/audit/hdfs/jn/SecurityAuth.audit

      /var/log/Bigdata/hdfs/*/

      Configure log collection paths for other components by referring to HDFS service logs.