Updated on 2024-10-08 GMT+08:00

Configuring Speculative Execution for MapReduce Tasks

Scenario

Clusters with hundreds or thousands of nodes are more prone to software and hardware issues, resulting in longer task execution times. This can cause tasks to be delayed until a malfunctioning device is fixed. In speculative execution, a task is allocated to multiple machines for running, and the node that completes the task first is selected. You can disable speculative execution for small clusters.

Procedure

Navigation path for setting parameters:

On the All Configurations page of the Yarn service, enter a parameter name in the search box. For details, see Modifying Cluster Service Configuration Parameters.

Parameter

Description

Default Value

mapreduce.map.speculative

Sets whether to execute multiple instances of some map tasks concurrently. true indicates that speculative execution is enabled.

false

mapreduce.reduce.speculative

Sets whether to execute multiple instances of some reduce tasks concurrently. true indicates that speculative execution is enabled.

false