Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Loader/ Common Issues About Loader/ An Error Is Reported When a Large Amount of Data Is Written to HDFS
Updated on 2024-05-29 GMT+08:00
View PDF
Share

An Error Is Reported When a Large Amount of Data Is Written to HDFS

Symptom

"NotReplicatedYet Exception: Not replicated yet" is occasionally reported when a large amount of data is written to HDFS.

Figure 1 Error message

Answer

The possible causes are as follows:

  1. The HDFS client sends a new block application to the NameNode. The NameNode does not process the application in time. As a result, the application times out.
  2. The incremental data reporting of DataNodes is too slow. As a result, NameNodes cannot allocate new blocks in a timely manner.

If this error occurs, the job will not become abnormal immediately. The job exception is notified only when the number of retry times exceeds the threshold. Increase the value of the HDFS parameter dfs.client.block.write.retries, for example, set the value to 10.