Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier)
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier)

Description

The system checks the host disk usage every 30 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated if the disk usage exceeds the threshold.

To change the threshold, choose System > Threshold Configuration.

This alarm is cleared when the host disk usage is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Auto Clear

12017

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

PartitionName

Specifies the disk partition for which the alarm is generated.

Trigger Condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

Service processes become unavailable.

Possible Causes

The disk configuration cannot meet service requirements. The disk usage reaches the upper limit.

Procedure

  1. Log in to MRS Manager and check whether the threshold is appropriate.

    1. The default threshold is 90%. You can change the threshold to meet service requirements.
      • If yes, go to 2.
      • If no, go to 1.b.
    2. Choose System > Threshold Configuration and change the alarm threshold based on the actual disk usage.
    3. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.

  2. Check whether the disk is a system disk.

    1. Go to the MRS cluster details page. In the alarm list on the alarm management tab page, click the row that contains the alarm. In the alarm details, view the host name and disk partition information.
    2. Log in to the node for which the alarm is generated.
    3. Run the df -h command to check the system disk partition usage. Check whether the disk is mounted to any of the following directories by using the disk partition name obtained in 2.a: /, /boot, /home, /opt, /tmp, /var, /var/log, /boot, and /srv/BigData.
      • If yes, the disk is a system disk. Then go to 3.a.
      • If no, the disk is not a system disk. Then go to 2.d.
    4. Run the df -h command to check the system disk partition usage. Determine the role of the disk based on the disk partition name obtained in 2.a.
    5. Check whether the disk is used by HDFS or Yarn.
      • If yes, expand the disk capacity for the Core node. Then go to 2.f.
      • If no, go to 4.
    6. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.

  3. Check whether large files are written to the disk.

    1. Run the find / -xdev -size +500M -exec ls -l {} \; command to view files larger than 500 MB on the node. Check whether such files are written to the disk.
      • If yes, go to 3.b.
      • If no, go to 4.
    2. Handle the large files and check whether the alarm is cleared 2 minutes later.
      • If yes, no further action is required.
      • If no, go to 4.
    3. Expand the disk capacity.
    4. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 4.

  4. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M engineers and send the collected logs.

Reference

None