Reducing Client Application Failure Rate
Scenario
When the network is unstable or the cluster I/O and CPU are overloaded, client applications might encounter running failures.
Configuration
Adjust the following parameters in the mapred-site.xml configuration file on the client to reduce the client application failure rate:
The mapred-site.xml configuration file is in the conf directory of the client installation path, for example, /opt/client/Yarn/config.
|
Parameter |
Description |
Default Value |
|---|---|---|
|
mapreduce.reduce.shuffle.max-host-failures |
Indicates the number of allowed failures of an MR task to read remote shuffle data in the Reduce process. When the number is set to be over 5, the client application failure rate can be reduced. |
5 |
|
mapreduce.client.submit.file.replication |
Indicates the backup of job files on HDFS. MR tasks are dependent on the job files during running. When the number of backups is set to be over 10, the client application failure rate can be reduced. |
10 |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
