Spark Streaming Task Fails but Is Displayed as Succeeded on the YARN Web UI
Symptom
In an MRS 3.3.0-LTS cluster, after a Spark Streaming task is submitted, the task fails on the corresponding node due to NodeAgent restart. However, the task status is displayed as succeeded on the YARN web UI.
Cause Analysis
By default, the parameter spark.streaming.stopGracefullyOnShutdown is set to false. This means that the system does not wait for the Spark ApplicationMaster to exit completely. As a result, the Spark ApplicationMaster task status may not be updated in time. In this way, the Spark ApplicationMaster task status may be displayed as failed when the task exits normally, or the status is displayed as succeeded when the task fails to exit.
Solution
When executing a Spark Streaming task, add the parameter --conf spark.streaming.stopGracefullyOnShutdown=true to the Spark task.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
