Updated on 2023-04-28 GMT+08:00

Why Cannot Exit the Driver Process?

Question

Why cannot exit the Driver process after running the yarn application -kill applicationID command to stop the Spark Streaming application?

Answer

Running the yarn application -kill applicationID command can only stop the SparkContext corresponding to Spark Streaming application, but cannot exit the current Driver process. If there are other permanent threads in the Driver process (for example, the spark shell is continually checking command input or Spark Streaming is continually reading data form data source), the Driver process will not be killed when the SparkContext is stopped. To exit the Driver process, you are advised to run the kill -9 pid command to kill the current Driver process by hand.