Insufficient Number of Replicas Is Reported During High Concurrent HDFS Writes

Updated on 2024-10-11 GMT+08:00

View PDF

Symptom

File writes to HDFS fail occasionally.

The operation log is as follows:

105 | INFO  | IPC Server handler 23 on 25000 | IPC Server handler 23 on 25000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.1.96:47728 Call#1461167 Retry#0 | Server.java:2278 
java.io.IOException: File /hive/warehouse/000000_0.835bf64f-4103 could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

Cause Analysis

HDFS has a reservation mechanism for file writing: each block to be written is 128 MB no matter whether the file is 10 MB or 1 GB. If a 10 MB file needs to be written, the file occupies 10 MB of the first block and about 118 MB space will be released. If a 1 GB file needs to be written, HDFS writes the file block by block and releases unused space after the file is written.

If there are a large number of files to be written concurrently, the disk space for reserved write blocks is insufficient. As a result, the file fails to be written.

Solution

Log in to the HDFS WebUI and go to the JMX page of the DataNode.
1. On the native HDFS page, choose Datanodes.
2. Locate the target DataNode and click the HTTP address to go to the DataNode details page.
3. Change datanode.html in url to jmx.
Search for the XceiverCount indicator. If the value of this indicator multiplied by the block size exceeds the DataNode disk capacity, the disk space reserved for block write is insufficient.
You can use either of the following methods to solve the problem:

Method 1: Reduce the service concurrency.

Method 2: Combine multiple files into one file to reduce the number of files to be written.

Parent topic: Using HDFS

Previous topic: Client Installed on a Node Outside the Cluster Fails to Upload Files Using hdfs

Next topic: HDFS Client Failed to Delete Overlong Directories

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot