Help Center/ MapReduce Service/ User Guide (ME-Abu Dhabi Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ System Configuration/ Configuring Interconnections/ Configuring Monitoring Indicator Data Dump
Updated on 2022-12-08 GMT+08:00
View PDF
Share

Configuring Monitoring Indicator Data Dump

Scenarios

The monitoring data reporting function writes the monitoring data collected in the system into a text file and uploads the file to a specified server in FTP or SFTP mode.

Before using this function, you need to perform related configurations on FusionInsight Manager.

Procedure

  1. Log in to FusionInsight Manager.
  2. Choose System > Interconnection > Upload Performance Data.
  3. Click the switch on the right of Upload Performance Data.

    Upload Performance Data is disabled by default. indicates the function is enabled.

  4. Set the upload parameters based on information provided in Table 1.

    Table 1 Uploading configuration parameters

    Parameter

    Description

    FTP IP Address Mode

    Specifies the server IP address mode. This parameter is mandatory.The value can be IPv4 or IPv6.

    FTP IP Address

    Specifies the FTP server for storing monitoring files after the monitoring indicator data is interconnected. This parameter is mandatory.

    FTP Port

    Specifies the port connected to the FTP server. This parameter is mandatory.

    FTP Username

    Specifies the username for logging in to the FTP server. This parameter is mandatory.

    FTP Password

    Specifies the password for logging in to the FTP server. This parameter is mandatory.

    Save Path

    Specifies the path for storing monitoring files on the FTP server. This parameter is mandatory.

    Dump Interval (s)

    Specifies the interval at which monitoring files are periodically stored on the FTP server, in seconds. This parameter is mandatory.

    Dump Mode

    Specifies the protocol used for sending monitoring files. This parameter is mandatory. The available values are FTP and SFTP. You are advised to use the SFTP mode based on SSH Version 2 (SSHv2). Otherwise, security risks may exist.

    SFTP Service Public Key

    Specifies the public key of the FTP server. This parameter is optional. This parameter is valid only when Dump Mode is set to SFTP.

  5. Click OK to complete the settings.

    In the scenario where the dump mode SFTP is used, if the public key of the SFTP service is empty, the system displays a security risk warning. Determine the security risk, and then save the configuration.

Data Format

After the configuration is complete, the monitoring data reporting function periodically writes monitoring data in the cluster to text files and reports the files to the corresponding FTP/SFTP service based on the configured reporting period.

  • Principles for Generating Monitoring Files
    • The monitoring indicators are written to files generated every 30, 60, and 300 seconds based on the indicator collection period.

      30s: indicates real-time indicators whose default collection period is 30s.

      60s: indicates real-time indicators whose default collection period is 60s.

      300s: indicates all other indicators except the ones whose period is 30s or 60s.

    • File name format: metirc_{Period}_{File creation time YYYYMMDDHHMMSS }.log

      For example: metric_60_20160908085915.log

      metric_300_20160908085613.log

  • Monitoring File Contents
    • Format of monitoring files:

      cluster ID |cluster name |indicator ID|collection time|collection host|unit|indicator value. Separate the fields from each other using vertical bars (|), for example:

      1|xx1|Host|Host|10000413|2022/06/18 10:05:00|189-66-254-146|KB/s|309.910
1|xx1|Host|Host|10000413|2022/06/18 10:05:00|189-66-254-152|KB/s|72.870
2|xx2|Host|Host|10000413|2022/06/18 10:05:00|189-66-254-163|KB/s|100.650

      Note: The actual files are not in the format.

    • Interval for uploading monitoring files:

      The interval for uploading monitoring files can be set using the Dump Interval (s) parameter on the page. Currently, the interval can be set to 30s to 300s. After the configuration is complete, the system periodically uploads files to the corresponding FTP/SFTP server at the specified interval.

  • Monitoring Indicator Description File
    • Indicator set file

      The indicator set file all-shown-metric-zh_CN contains detailed information about all indicators. After obtaining the indicator IDs from the files reported by the third-party system, you can query details about the indicators from the indicator set file.

      Location of the indicator set file:

      Active and standby OMS nodes: {FusionInsight installation path} /om-server/om/etc/om/all-shown-metric-zh_CN

      Contents of the indicator set file:

      Real-time indicator ID, 5-minute indicator ID, indicator name, indicator collection period (second), whether to collect data by default, service to which the indicator belongs, and role to which the indicator belongs
00101,10.000101,JobHistoryServer non-heap memory usage,30,false,Mapreduce,JobHistoryServer
00102,10.000102,JobHistoryServer Non-heap memory allocation volume,30,false,Mapreduce,JobHistoryServer
00103,10.000103,JobHistoryServer heap memory usage,30,false,Mapreduce,JobHistoryServer
00104,10.000104,JobHistoryServer heap memory allocation volume,30,false,Mapreduce,JobHistoryServer
00105, 10.000105,Number of blocked threads,30,false,Mapreduce,JobHistoryServer
00106,10.000106,Number of running threads,30,false,Mapreduce,JobHistoryServer
00107,10.000107,GC time,30,false,Mapreduce,JobHistoryServer
00110,10.00011,JobHistoryServer CPU usage,30,false,Mapreduce,JobHistoryServer
Real-Time Metric ID,5-Minute Metric ID,Metric Name,Metric Collection Period (s),Collected by Default,Service Belonged To,Role Belonged To 00101,10000101,Used Non Heap Memory of JobHistoryServer,60,false,Mapreduce,JobHistoryServer                                           00102,10000102,Allocated Non Heap Memory of JobHistoryServer,60,false,Mapreduce,JobHistoryServer                                   00103,10000103,Used Heap Memory of JobHistoryServer,60,false,Mapreduce,JobHistoryServer                                                  00104,10000104,Allocated Heap Memory of JobHistoryServer,60,false,Mapreduce,JobHistoryServer                                    00105,10000105,Blocked Threads,30,false,Mapreduce,JobHistoryServer                                                                     00106,10000106,Running Threads,30,false,Mapreduce,JobHistoryServer                                                                00107,10000107,GC Time,60,false,Mapreduce,JobHistoryServer
    • Field description of critical indicators

      Real-Time Metric ID: indicates the ID of the indicator whose collection period is 30s or 60s.

      5-Minute Metric: The ID of a 5-minute (300s) indicator.

      Metric Collection Period (s): Real-time collection period of indicators. The value can be 30 or 60.

      Service Belonged To: Name of the service to which an indicator belongs, indicating the service type, for example, HDFS and HBase.

      Role to which an indicator belongs: indicates the name (type) of the role to which an indicator belongs, for example, JobServer or RegionServer.

    • Description

      For metrics whose collection period is 30s/60s, you can find the corresponding metric description by referring to the first column, that is, Real-Time Metric ID.

      For metrics whose collection period is 300s, you can find the corresponding metric description by referring to the second column, that is, 5-Minute Metric.