On this page

Show all

Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-14011 DataNode Data Directory Is Not Configured Properly

ALM-14011 DataNode Data Directory Is Not Configured Properly

Updated on 2022-08-12 GMT+08:00

View PDF

Description

The DataNode parameter dfs.datanode.data.dir specifies DataNode data directories. This alarm is generated when a configured data directory cannot be created, a data directory uses the same disk as other critical directories in the system, or multiple directories use the same disk immediately.

This alarm is cleared when the DataNode data directory is configured properly and this DataNode for which the alarm is generated is restarted.

Attribute

Alarm ID	Alarm Severity	Automatically Cleared
14011	Major	Yes

Parameters

Name	Meaning
Source	Specifies the cluster for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.

Impact on the System

If the DataNode data directory is mounted to the root directory or a critical directory, the disk space of the root directory or critical directory will be used up after long time running and the system will be faulty.

If the DataNode data directory is not configured properly, HDFS performance will deteriorate.

Possible Causes

The DataNode data directory fails to be created.
The DataNode data directory uses the same disk with critical directories, such as / or /boot.
Multiple directories in the DataNode data directory use the same disk.

Procedure

Check the alarm cause and information about the DataNode for which the alarm is generated.

On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. In the alarm list, click the alarm.
In HostName of Location, obtain the host name of the DataNode for which the alarm is generated.

Delete directories that do not comply with the disk plan from the DataNode data directory.

Choose Cluster > Name of the desired cluster > Services > HDFS > Instance. In the instance list, click the DataNode instance on the node for which the alarm is generated.
Click Instance Configurations and view the value of the DataNode parameter dfs.datanode.data.dir.
Check whether all DataNode data directories are consistent with the disk plan.
- If yes, go to 6.
- If no, go to 9.
Modify the DataNode parameter dfs.datanode.data.dir and delete the incorrect directories.
Choose Cluster > Name of the desired cluster > Services > HDFS > Instance and restart the DataNode instance.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Log in to the DataNode for which the alarm is generated as user root.
- If the alarm cause is "The DataNode data directory fails to be created", go to 10.
- If the alarm cause is "The DataNode data directory uses the same disk with critical directories, such / or /boot", go to 17.
- If the alarm cause is "Multiple directories in the DataNode data directory uses the same disk", go to 21.

Check whether the DataNode data directory fails to be created.

Run the su - omm command to switch to user omm.
Run the ls command to check whether the directories exist in the DataNode data directory.
- If yes, go to 26.
- If no, go to 12.
Run the mkdir data directory command to create the directory and check whether the directory can be successfully created.
- If yes, go to 24.
- If no, go to 13.
On the FusionInsight Manager portal, choose O&M > Alarm > Alarms to check whether alarm ALM-12017 Insufficient Disk Capacity exists.
- If yes, go to 14.
- If no, go to 15.
Adjust the disk capacity and check whether alarm ALM-12017 Insufficient Disk Capacity is cleared. For details, see ALM-12017 Insufficient Disk Capacity.
- If yes, go to 12.
- If no, go to 15.
Check whether user omm has the rwx or x permission of all the upper-layer directories of the directory. (For example, for /tmp/abc/, user omm has the x permission for directory tmp and the rwx permission for directory abc.)
- If yes, go to 24.
- If no, go to 16.
Run the chmod u+rwx path or chmod u+x path command as user root to assign the rwx or x permission of these directories to user omm. Then go to 12.

Check whether the DataNode data directory use the same disk as other critical directories in the system.

Run the df command to obtain the disk mounting information of each directory in the DataNode data directory.
Check whether the directories mounted to the disk are critical directories, such as / or /boot.
- If yes, go to 19.
- If no, go to 24.
Change the value of the DataNode parameter dfs.datanode.data.dir and delete the directories that use the same disk as critical directories.
Go to 24.

Check whether multiple directories in the DataNode data directory use the same disk.

Run the df command to obtain the disk mounting information of each directory in the DataNode data directory. Record the mounted directory in the command output.
Modify the DataNode node parameters dfs.datanode.data.dir to reserve only one directory among the directories that mounted to the same disk directory.
Go to 24.

Restart the DataNode and check whether the alarm is cleared.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Instance and restart the DataNode instance
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 26.

Collect fault information.

On the FusionInsight Manager portal, choose O&M > Log > Download.
Select HDFS in the required cluster from the Service.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None

Parent topic: Alarm Reference (Applicable to MRS 3.x)

Previous topic: ALM-14010 NameService Service Is Abnormal

Next topic: ALM-14012 JournalNode Is Out of Synchronization

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot