Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-14029 Number of Blocks in a Replica Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-14029 Number of Blocks in a Replica Exceeds the Threshold

Alarm Description

The system checks the number of blocks in a single replica every four hours and compares the number with the threshold. There is a threshold for the number of blocks in a single replica. This alarm is generated when the actual number of blocks in a single replica exceeds the threshold.

This alarm is cleared when the number of blocks to be supplemented is less than the threshold.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

14029

Minor

Yes

Alarm Parameters

Parameter

Description

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

NameServiceName

Specifies the NameService for which the alarm was generated.

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

Replica data is prone to be lost when a node is faulty. Too many files of a single replica affect the security of the HDFS file system.

Possible Causes

  • The DataNode is faulty.
  • The disk is faulty.
  • Files are written to a single replica.

Handling Procedure

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, check whether alarm ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold is generated.

    • If yes, go to 2.
    • If no, go to 3.

  2. Rectify the fault according to the handling procedure of ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold. In the next detection period, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 3.

  3. Check whether files of a single replica have been written into the service.

    • If yes, go to 4.
    • If no, go to 7.

  4. Log in to the HDFS client as user root. The user password is defined by the user before the installation. Contact the MRS cluster administrator to obtain the password. Run the following commands:

    • Security mode:

      cd Client installation directory

      source bigdata_env

      kinit hdfs

    • Normal mode:

      su - omm

      cd Client installation directory

      source bigdata_env

  5. Run the following command on the client node to increase the number of replicas for a single replica file:

    hdfs dfs -setrep -w file replica number file name or file path

  6. In the next detection period, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 7.

Collect the fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None