Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using MapReduce/ Common Issues About MapReduce/ How Do I Handle the Problem that MapReduce Task Has No Progress for a Long Time?
Updated on 2024-10-08 GMT+08:00
View PDF
Share

How Do I Handle the Problem that MapReduce Task Has No Progress for a Long Time?

Question

The MapReduce task has no progress for a long time.

Answer

Generally, this is caused by insufficient memory. If the memory is small, it takes a long time to copy the map output.

To reduce the waiting time, increase the heap memory.

You can optimize task configuration based on the number of mappers and the data size of each mapper. Optimize the following parameters in the Client installation path/Yarn/config/mapred-site.xml file based on the size of the input data:

  • mapreduce.reduce.memory.mb
  • mapreduce.reduce.java.opts

For example, if the data size of 10 mappers is 5 GB, the ideal heap memory is 1.5 GB. Increase the heap memory as the data size increases.