ALM-14011 HDFS DataNode Data Directory Is Not Configured Properly (For MRS 2.x or Earlier)

Description

The DataNode parameter dfs.datanode.data.dir specifies the DataNode data directory. This alarm is generated in any of the following scenarios:

• A configured data directory cannot be created.
• A data directory uses the same disk as other critical directories in the system.
• Multiple directories use the same disk.

This alarm is cleared when the DataNode data directory is configured properly and this DataNode is restarted.

Attribute

Parameters

Impact on the System

If the DataNode data directory is mounted on critical directories such as the root directory, the disk space of the root directory will be used up after running for a long time. This causes a system fault.

If the DataNode data directory is not configured properly, HDFS performance will deteriorate.

Possible Causes

• The DataNode data directory fails to be created.
• The DataNode data directory uses the same disk as critical directories, such as / or /boot.
• Multiple directories in the DataNode data directory use the same disk.

Procedure

1. Check the alarm cause and information about the DataNode for which the alarm is generated.
   1. On the MRS cluster details page, click Alarms. In the alarm list, click the alarm.
   2. In the Alarm Details area, view Alarm Cause to obtain the cause of the alarm. In HostName of Location, obtain the host name of the DataNode for which the alarm is generated.

2. Delete directories that do not comply with the disk plan from the DataNode data directory.
   1. Choose Components > HDFS > Instances. In the instance list, click the DataNode instance on the node for which the alarm is generated.
   2. Click Instance Configuration and view the value of the DataNode parameter dfs.datanode.data.dir.
   3. Check whether all DataNode data directories are consistent with the disk plan.
      • If yes, go to 2.d.
      • If no, go to 2.g.
   4. Modify the DataNode parameter dfs.datanode.data.dir and delete the incorrect directories.
   5. Choose Components > HDFS > Instances to restart the DataNode instance.
   6. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.g.
   7. Log in to the DataNode for which the alarm is generated.
      • If the alarm cause is "The DataNode data directory fails to be created", go to 3.a.
      • If the alarm cause is "The DataNode data directory uses the same disk as critical directories, such / or /boot", go to 4.a.
      • If the alarm cause is "Multiple directories in the DataNode data directory use the same disk", go to 5.a.

3. Check whether the DataNode data directory fails to be created.
   1. Run the following commands to switch the user:

      sudo su - root

      su - omm

   2. Run the ls command to check whether the directories exist in the DataNode data directory.
      • If yes, go to 7.
      • If no, go to 3.c.
   3. Run the mkdir data directory command to create a directory and check whether the directory is successfully created.
      • If yes, go to 6.a.
      • If no, go to 3.d.
   4. Click Alarms to check whether alarm ALM-12017 Insufficient Disk Capacity exists.
      • If yes, go to 3.e.
      • If no, go to 3.f.
   5. Adjust the disk capacity and check whether alarm ALM-12017 Insufficient Disk Capacity is cleared. For details, see ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier).
      • If yes, go to ALM-12017 Insufficient Disk Capacity (For MRS 2.x or Earlier).
      • If no, go to 7.
   6. Check whether user omm has the rwx or x permission of all the upper-layer directories of the directory. (For example, for /tmp/abc/, user omm has the x permission for directory tmp and the rwx permission for directory abc.)
      • If yes, go to 6.a.
      • If no, go to 3.g.
   7. Run the chmod u+rwx path or chmod u+x path command as the root user to add the rwx or x permission to the paths. Then, go to 3.c.

4. Check whether the DataNode data directory uses the same disk as other critical directories in the system.
   1. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory.
   2. Check whether the directories mounted to the disk are critical directories, such as / or /boot.
      • If yes, go to 4.c.
      • If no, go to 6.a.
   3. Change the value of the DataNode parameter dfs.datanode.data.dir and delete the directories that use the same disk as critical directories.
   4. Go to 6.a.

5. Check whether multiple directories in the DataNode data directory use the same disk.
   1. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory. Record the mounted directory in the command output.
   2. Modify the DataNode node parameter dfs.datanode.data.dir to reserve one of the directories mounted on the same disk directory.
   3. Go to 6.a.

6. Restart the DataNode and check whether the alarm is cleared.
   1. Choose Components > HDFS > Instances to restart the DataNode instance.
   2. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 7.

7. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact the O&M engineers and send the collected logs.

Reference

None