Help Center/ MapReduce Service/ User Guide/ Alarm Reference (Applicable to MRS 3.x)/ ALM-45481 KuduTserver Has Full Disks
Updated on 2024-05-07 GMT+08:00
View PDF

ALM-45481 KuduTserver Has Full Disks

The system checks Kudu disk metrics every 60 seconds. This alarm is generated when the number of fully occupied disks for a Tserver is not 0.

This alarm is automatically cleared when the number of fully occupied disks for the Tserver becomes 0.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

45481

Major

Yes

Alarm Parameters

Type

Parameter

Description

Location Information

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Additional Information

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

Users cannot use the Kudu service properly.

Possible Causes

The disk configuration cannot meet service requirements, and the disk space used by the Kudu service is insufficient.

Handling Procedure

Check the disk capacity and delete unnecessary files and data tables.

  1. Log in to FusionInsight Manager. In the alarm list, click the drop-down arrow in the row that contains the alarm, and view the role name and the IP address of the hostname in Location.
  2. Log in to the node for which the alarm is generated as user root and run the df -h command to check the mount directory where the disk usage is 100%.
  3. Run the find / -xdev -size +500M; command to view files larger than 500 MB in the directory obtained in 2. Check whether such files are mistakenly written to the directory.

    • If yes, go to 4.
    • If no, go to 5.

  4. Delete the mistakenly written files and check whether the alarm is cleared 2 minutes later.

    • If yes, no further action is required.
    • If no, go to 5.

  5. Check whether Kudu contains unnecessary data tables.

    • If yes, go to 6.
    • If no, go to 8.

  6. Use kudu table delete <master_addresses> <table_name> [-nomodify_external_catalogs] to delete unnecessary tables.
  7. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 8.

  8. Contact the disk administrator to expand the disk capacity.
  9. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 10.

Collect fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, select Kudu for the target cluster, and click OK.
  3. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None