Configuring Heap Memory Parameters for Spark Executor
Scenarios
When the executor off-heap memory is too small, or processes with higher priority preempt resources, the physical memory usage will exceed the maximal value. To prevent the physical memory usage from exceeding, set the following parameter.
Configuration
- Install the Spark client.
For details, see Installing a Client.
- Log in to the Spark client node as the client installation user.
Modify the following parameter in the {Client installation directory}/Spark/spark/conf/spark-defaults.conf configuration file on the client or set the parameter using --conf when submitting an application.
Table 1 Parameter description Parameter
Description
Example Value
spark.executor.memoryOverhead
Indicates amount of off-heap memory allocated to each executor, measured in MB. Increasing the value of this parameter prevents the physical memory usage from exceeding the physical memory limits.
The value is calculated based on max(384, Executor – Memory x 0.1). The minimal value is 384.
1024
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot