ALM-12017 Insufficient Disk Capacity

Description

The system checks the host disk usage of the system every 30 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold, this alarm is generated when the host disk usage exceeds the specified threshold.

When the Trigger Count is 1, this alarm is cleared when the usage of a host disk partition is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the usage of a host disk partition is less than or equal to 90% of the threshold.

Attribute

Parameters

Impact on the System

Service failure: If you need to modify or use data on the disk when the disk capacity is insufficient, the job may fail.

Possible Causes

• The alarm threshold is incorrect.
• Disk configuration of the server cannot meet service requirements.

Procedure

Check whether the alarm threshold is appropriate.

1. Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > Disk > Disk Usage and check whether the threshold (configurable, 90% by default) is appropriate.
   • If yes, go to 2.
   • If no, go to 4.

2. Choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > Disk > Disk Usage and click Modify in the Operation column to change the alarm threshold based on site requirements. As shown in Figure 1:
   Figure 1 Setting an alarm threshold
   

3. After 2 minutes, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 4.

Check whether the disk usage reaches the upper limit.

4. In the alarm list on FusionInsight Manager, click in the row where the alarm is located to view the alarm host name and disk partition information in the alarm details.
5. Log in to the node where the alarm is generated as user root.
6. Run the df -lmPT | awk '$2 != "iso9660"' | grep '^/dev/' | awk '{"readlink -m "$1 | getline real }{$1=real; print $0}' | sort -u -k 1,1 command to check the system disk partition usage. Check whether the disk is mounted to the following directories based on the disk partition name obtained in 4: /, /opt, /tmp, /var, /var/log, and /srv/BigData(can be customized).
   • If yes, the disk is a system disk. Then go to 10.
   • If no, the disk is not a system disk. Then go to 7.

7. Run the df -lmPT | awk '$2 != "iso9660"' | grep '^/dev/' | awk '{"readlink -m "$1 | getline real }{$1=real; print $0}' | sort -u -k 1,1 command to check the system disk partition usage. Determine the role of the disk based on the disk partition name obtained in 4.
8. Check the disk service.
   In MRS, check whether the disk service is HDFS, Yarn, Kafka, Supervisor.

   • If yes, adjust the capacity. Then go to 9.
   • If no, go to 12.

9. After 2 minutes, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 12.

10. Run the find / -xdev -size +500M -execls -l {} \; command to check whether a file larger than 500 MB exists on the node and disk.
    • If yes, go to 11.
    • If no, go to 12.

11. Handle the large file and check whether the alarm is cleared 2 minutes later.
    • If yes, no further action is required.
    • If no, go to 12.

12. Contact the system administrator to expand the disk capacity.
13. After 2 minutes, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Collect fault information.

14. On FusionInsight Manager, choose O&M > Log > Download.
15. Select OMS from the Service and click OK.
16. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
17. Contact the O&M personnel and send the collected log information.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None