Why Cannot Exit the Driver Process?
Question
Why cannot exit the Driver process after running the yarn application -kill applicationID command to stop the Spark Streaming application?
Answer
Running the yarn application -kill applicationID command can only stop the SparkContext corresponding to Spark Streaming application, but cannot exit the current Driver process. If there are other permanent threads in the Driver process (for example, the spark shell is continually checking command input or Spark Streaming is continually reading data form data source), the Driver process will not be killed when the SparkContext is stopped. To exit the Driver process, you are advised to run the kill -9 pid command to kill the current Driver process by hand.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
