Configuring Spark Dynamic Resource Scheduling in YARN Mode

Scenarios

Resources are a key factor that affects Spark execution efficiency. Allocating multiple executors to a long-running service, such as the JDBCServer, without tasks can result in improper scheduling and wasted resources if there are insufficient resources available for other applications.

Dynamic resource scheduling can add or remove executors of applications in real time based on the task load. In this way, resources are dynamically scheduled to applications.

Dynamic resource scheduling must be used in conjunction with the external shuffle service. Otherwise, shuffle data will be lost when executors are removed.

Procedure

1. Before enabling dynamic resource scheduling, ensure that the external shuffle service has been configured for Spark. For details, see Using the External Shuffle Service to Improve Performance.

   In Spark, the external shuffle service is a process independent of the executors. It is used to optimize the performance and stability of shuffle operations. It separates the management of shuffle data from the executors, preventing shuffle data loss when executors exit and reducing resource contention.

2. Log in to FusionInsight Manager, choose Cluster > Services > Spark2x, and click Configurations and then All Configurations. Enter the spark.dynamicAllocation.enabled parameter name in the search box and set it to true to enable dynamic resource scheduling.
   The following are optional configurations, as shown in Table 1. You can adjust them based on your actual business requirements.

   Table 1 Parameters for dynamic resource scheduling

   Configuration Item	Description	Example Value
   spark.dynamicAllocation.minExecutors	Minimum number of executors to maintain if dynamic allocation is enabled. When dynamic allocation is enabled, Spark automatically adjusts the number of executors based on the workload. However, at least this specified number of executors will be retained to ensure basic compute resources. It is recommended to set this parameter to an appropriate value based on your minimum workload, for example, 1.	0
   spark.dynamicAllocation.initialExecutors	Initial number of executors to run if Spark's dynamic allocation is enabled. During application startup, executors are pre-allocated based on the estimated workload to prevent task delays caused by insufficient resources in the initial stage. The value must be greater than or equal to spark.dynamicAllocation.minExecutors and less than or equal to spark.dynamicAllocation.minExecutors.	0
   spark.dynamicAllocation.maxExecutors	Maximum number of executors that Spark's dynamic allocation feature can allocate for a given application. When dynamic allocation is enabled, Spark automatically adjusts the number of executors based on the workload, but the total number of executors will not exceed the value of this parameter. This helps control the upper limit of resource usage, prevent resource monopoly, and protect other applications in the cluster.	2048
   spark.dynamicAllocation.schedulerBacklogTimeout	How long the scheduler waits for a backlog of pending tasks to accumulate before requesting new executors when Spark's dynamic allocation is enabled. If dynamic allocation is enabled and there have been pending tasks backlogged for more than this duration, Spark will request additional executors to accelerate processing.	1s
   spark.dynamicAllocation.sustainedSchedulerBacklogTimeout	Time interval after which Spark will continue to request additional executors if there remains a sustained backlog of pending tasks in the scheduler. If pending tasks remain after the initial executor request, Spark will continue to request additional executors at intervals defined by this parameter until the configured maximum number of executors is reached. It is recommended to set this parameter to a value greater than the initial timeout interval.	1s
   spark.dynamicAllocation.executorIdleTimeout	The duration after which an idle executor (one with no running tasks) will be removed. If dynamic allocation is enabled and an executor has been idle for more than this duration, Spark will release the executor to reclaim cluster resources, avoiding resource waste.	60s
   spark.dynamicAllocation.cachedExecutorIdleTimeout	Maximum amount of time an executor that has cached data blocks can remain idle before it is removed. Unlike regular executors, those holding cached data blocks can remain idle for a longer duration without being removed. This behavior helps preserve cached data, reducing the overhead of re-computation.	JDBCServer: 2147483647s IndexServer: 2147483647s SparkResource: 120