ALM-14011 DataNode Data Directory Is Not Configured Properly

Description

The DataNode parameter dfs.datanode.data.dir specifies DataNode data directories. This alarm is generated when a configured data directory cannot be created, a data directory uses the same disk as other critical directories in the system, or multiple directories use the same disk immediately.

This alarm is cleared when the DataNode data directory is configured properly and this DataNode for which the alarm is generated is restarted.

Attribute

Parameters

Impact on the System

If the DataNode data directory is mounted to the root directory or a critical directory, the disk space of the root directory or critical directory will be used up after long time running and the system will be faulty.

If the DataNode data directory is not configured properly, HDFS performance will deteriorate.

Possible Causes

• The DataNode data directory fails to be created.
• The DataNode data directory uses the same disk with critical directories, such as / or /boot.
• Multiple directories in the DataNode data directory use the same disk.

Procedure

Check the alarm cause and information about the DataNode for which the alarm is generated.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. In the alarm list, click the alarm.
2. In HostName of Location, obtain the host name of the DataNode for which the alarm is generated.

Delete directories that do not comply with the disk plan from the DataNode data directory.

3. Choose Cluster > Name of the desired cluster > Services > HDFS > Instance. In the instance list, click the DataNode instance on the node for which the alarm is generated.
4. Click Instance Configurations and view the value of the DataNode parameter dfs.datanode.data.dir.
5. Check whether all DataNode data directories are consistent with the disk plan.
   • If yes, go to 6.
   • If no, go to 9.

6. Modify the DataNode parameter dfs.datanode.data.dir and delete the incorrect directories.
7. Choose Cluster > Name of the desired cluster > Services > HDFS > Instance and restart the DataNode instance.
   

   Services may be affected or interrupted during the restart. You are advised to perform this operation during off-peak hours.

8. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

9. Log in to the DataNode for which the alarm is generated as user root.
   • If the alarm cause is "The DataNode data directory fails to be created", go to 10.
   • If the alarm cause is "The DataNode data directory uses the same disk with critical directories, such / or /boot", go to 17.
   • If the alarm cause is "Multiple directories in the DataNode data directory uses the same disk", go to 21.

Check whether the DataNode data directory fails to be created.

10. Run the su - omm command to switch to user omm.
11. Run the ls command to check whether the directories exist in the DataNode data directory.
    • If yes, go to 26.
    • If no, go to 12.

12. Run the mkdir data directory command to create the directory and check whether the directory can be successfully created.
    • If yes, go to 24.
    • If no, go to 13.

13. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms to check whether alarm ALM-12017 Insufficient Disk Capacity exists.
    • If yes, go to 14.
    • If no, go to 15.

14. Adjust the disk capacity and check whether alarm ALM-12017 Insufficient Disk Capacity is cleared. For details, see ALM-12017 Insufficient Disk Capacity.
    • If yes, go to 12.
    • If no, go to 15.

15. Check whether user omm has the rwx or x permission of all the upper-layer directories of the directory. (For example, for /tmp/abc/, user omm has the x permission for directory tmp and the rwx permission for directory abc.)
    • If yes, go to 24.
    • If no, go to 16.

16. Run the chmod u+rwx path or chmod u+x path command as user root to assign the rwx or x permission of these directories to user omm. Then go to 12.

Check whether the DataNode data directory use the same disk as other critical directories in the system.

17. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory.
18. Check whether the directories mounted to the disk are critical directories, such as / or /boot.
    • If yes, go to 19.
    • If no, go to 24.

19. Change the value of the DataNode parameter dfs.datanode.data.dir and delete the directories that use the same disk as critical directories.
20. Go to 24.

Check whether multiple directories in the DataNode data directory use the same disk.

21. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory. Record the mounted directory in the command output.
22. Modify the DataNode node parameters dfs.datanode.data.dir to reserve only one directory among the directories that mounted to the same disk directory.
23. Go to 24.

Restart the DataNode and check whether the alarm is cleared.

24. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Instance and restart the DataNode instance.
    

    Services may be affected or interrupted during the restart. You are advised to perform this operation during off-peak hours.

25. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 26.

Collect fault information.

26. On the FusionInsight Manager portal, choose O&M > Log > Download.
27. Select HDFS in the required cluster from the Service.
28. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
29. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None