Enabling a Spark Executor to Execute Custom Code When Exiting

Scenarios

In Spark, an executor is a process that executes specific tasks on worker nodes in a cluster. When a Spark executor is terminated, it must perform a series of cleanup operations. These operations typically include releasing allocated memory, closing open file handles, and disconnecting network connections. These cleanup operations may take a long time due to a variety of factors. To prevent cleanup operations from being blocked indefinitely, you can configure a maximum timeout interval. If cleanup operations exceed the configured timeout interval, the system will forcibly terminate the cleanup process to ensure that cluster resources are promptly released and reallocated efficiently.

Notes and Constraints

This section applies only to MRS 3.2.0-LTS or later.

Configuration Parameters

1. Install the Spark client.

   For details, see Installing a Client.

2. Log in to the Spark client node as the client installation user.

   Modify the following parameters in the {Client installation directory}/Spark/spark/conf/spark-defaults.conf file on the Spark client.

   Parameter	Description	Example Value
   spark.executor.execute.shutdown.cleaner	Controls the resource cleanup behavior during the shutdown of a Spark executor. This parameter determines whether to perform additional cleanup operations (such as deleting temporary files and releasing external resources) when an executor exits. Enabling this ensures that the cluster environment is released promptly, improving overall system efficiency. true: Cleanup operations are performed when an executor exits. false: No cleanup operation is performed. The default value is false.	true
   spark.executor.execute.shutdown.cleaner.max.timeout	Specifies the maximum timeout interval for resource cleanup during the shutdown of a Spark executor. This parameter specifies the maximum waiting time allowed for an executor to perform cleanup operations (such as deleting temporary files and releasing external resources) before exiting. By setting this timeout, Spark ensures that the process of shutting down an executor does not get indefinitely blocked by ongoing cleanup operations.	240s