ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier)

Description

The system checks the host disk usage every 30 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated if the disk usage exceeds the threshold.

To change the threshold, choose System > Threshold Configuration.

This alarm is cleared when the host disk usage is less than or equal to the threshold.

Attribute

Parameters

Impact on the System

Service processes become unavailable.

Possible Causes

The disk configuration cannot meet service requirements. The disk usage reaches the upper limit.

Procedure

1. Log in to MRS Manager and check whether the threshold is appropriate.
   1. The default threshold is 90%. You can change the threshold to meet service requirements.
      • If yes, go to 2.
      • If no, go to 1.b.
   2. Choose System > Threshold Configuration and change the alarm threshold based on the actual disk usage.
   3. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.

2. Check whether the disk is a system disk.
   1. Go to the MRS cluster details page. In the alarm list on the alarm management tab page, click the row that contains the alarm. In the alarm details, view the host name and disk partition information.
   2. Log in to the node for which the alarm is generated.
   3. Run the df -h command to check the system disk partition usage. Check whether the disk is mounted to any of the following directories by using the disk partition name obtained in 2.a: /, /boot, /home, /opt, /tmp, /var, /var/log, /boot, and /srv/BigData.
      • If yes, the disk is a system disk. Then go to 3.a.
      • If no, the disk is not a system disk. Then go to 2.d.
   4. Run the df -h command to check the system disk partition usage. Determine the role of the disk based on the disk partition name obtained in 2.a.
   5. Check whether the disk is used by HDFS or Yarn.
      • If yes, expand the disk capacity for the Core node. Then go to 2.f.
      • If no, go to 4.
   6. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.

3. Check whether large files are written to the disk.
   1. Run the find / -xdev -size +500M -exec ls -l {} \; command to view files larger than 500 MB on the node. Check whether such files are written to the disk.
      • If yes, go to 3.b.
      • If no, go to 4.
   2. Handle the large files and check whether the alarm is cleared 2 minutes later.
      • If yes, no further action is required.
      • If no, go to 4.
   3. Expand the disk capacity.
   4. Wait 2 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 4.

4. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact the O&M engineers and send the collected logs.

Reference

None