Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ MRS Cluster Component Operation Gudie/ Using Spark2x/ Common Issues About Spark2x/ Spark Core/ Why Cannot Exit the Driver Process?
Updated on 2022-08-12 GMT+08:00
View PDF
Share

Why Cannot Exit the Driver Process?

Question

Why cannot exit the Driver process after running the yarn application -kill applicationID command to stop the Spark Streaming application?

Answer

Running the yarn application -kill applicationID command can only stop the SparkContext corresponding to Spark Streaming application, but cannot exit the current Driver process. If there are other permanent threads in the Driver process (for example, the spark shell is continually checking command input or Spark Streaming is continually reading data form data source), the Driver process will not be killed when the SparkContext is stopped. To exit the Driver process, you are advised to run the kill -9 pid command to kill the current Driver process by hand.

Parent topic: Spark Core