Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Spark/Spark2x/ Spark Core Performance Tuning/ Adjusting Spark Core Process Parameters
Updated on 2026-08-29 GMT+08:00
View PDF
Share

Adjusting Spark Core Process Parameters

Scenarios

Spark on YARN mode relies on three core processes: the Driver, the ApplicationMaster, and the Executor. The Driver and Executor manage task scheduling and execution, while the ApplicationMaster manages container lifecycles (allocation and termination).

Therefore, Driver and Executor configurations significantly impact the performance of Spark applications. Apply the following tuning strategies to optimize Spark cluster performance:

Procedure

  1. Configure the driver memory.

    The driver schedules tasks and communicates with the executor and the ApplicationMaster. Add driver memory when the number and parallelism level of the tasks increases.

    You can configure the driver memory based on the number of the tasks.

    • Set spark.driver.memory in spark-defaults.conf to a proper value.
    • Add the --driver-memory MEM parameter to configure the memory when using the spark-submit command.

  2. Configure the number of the executors.

    One core in an executor can run one task at the same time. Therefore, more tasks can be processed at the same time if you increase the number of the executors. You can add the number of the executors to increase the efficiency if resources are sufficient.

    • Set spark.executor.instance in spark-defaults.conf or SPARK_EXECUTOR_INSTANCES in spark-env.sh to a proper value.
    • Add the --num-executors NUM parameter to configure the number of the executors when using the spark-submit command.

  3. Configure the number of the executor cores.

    Multiple cores in an executor can run multiple tasks at the same time, which increases the task concurrency. However, because all cores share the memory of an executor, you need to balance the memory and the number of cores.

    • Set spark.executor.cores in spark-defaults.conf or SPARK_EXECUTOR_CORES in spark-env.sh to a proper value.
    • When you run the spark-submit command, add the --executor-cores NUM parameter to set the number of executor cores.

  4. Configure the executor memory.

    The executor memory is used for task execution and communication. You can increase the memory for a big task that needs more resources, and reduce the memory to increase the concurrency level for a small task that runs fast.

    • Set spark.executor.memory in spark-defaults.conf or SPARK_EXECUTOR_MEMORY in spark-env.sh to a proper value.
    • When you run the spark-submit command, add the --executor-memory MEM parameter to set the memory.

Tuning Example

  • In a Spark WordCount workload processing a 1.6 TB dataset across 250 executors, the following configurations apply:

    The execution fails under the default configuration, and the Futures timed out and OOM errors occur.

    However each task of WordCount is small and runs fast, the amount of data is large and the tasks are too many. Therefore the objects on the driver end become huge when there are many tasks. Besides the fact that the executor communicates with the driver once each task is finished, the problem of disconnection between processes caused by insufficient memory occurs.

    The application executes successfully once the driver memory is allocated at 4 GB.

  • When executing TPC-DS benchmarks via the JDBCServer using default configuration parameters, the system encounters the "Executor Lost" error. To ensure stable and successful task execution, the cluster parameters must be tuned to a minimum of 30 GB driver memory, and 125 executors each with 2 executor cores and 6 GB of executor memory.