Configuring Heap Memory Parameters for Spark Executor

Scenarios

When the executor off-heap memory is too small, or processes with higher priority preempt resources, the physical memory usage will exceed the maximal value. To prevent the physical memory usage from exceeding, set the following parameter.

Configuration

1. Install the Spark client.

   For details, see Installing a Client.

2. Log in to the Spark client node as the client installation user.

   Modify the following parameter in the {Client installation directory}/Spark/spark/conf/spark-defaults.conf configuration file on the client or set the parameter using --conf when submitting an application.

   Table 1 Parameter description

   Parameter	Description	Example Value
   spark.executor.memoryOverhead	Indicates amount of off-heap memory allocated to each executor, measured in MB. Increasing the value of this parameter prevents the physical memory usage from exceeding the physical memory limits. The value is calculated based on max(384, Executor – Memory x 0.1). The minimal value is 384.	1024