Help Center/ MapReduce Service/ Troubleshooting/ Using Spark/ Spark Streaming Task Failed but Status Shows as Successful in Yarn WebUI
Updated on 2025-08-21 GMT+08:00

Spark Streaming Task Failed but Status Shows as Successful in Yarn WebUI

Symptom

In an MRS 3.3.0-LTS cluster, after a Spark Streaming task is submitted, the task on the corresponding node fails due to NodeAgent restart. However, the task status on the Yarn WebUI is succeeded.

Cause Analysis

When Spark AM exits, the default value of spark.streaming.stopGracefullyOnShutdown is false. Therefore, the system does not wait for the AM to exit normally. Therefore, there is a possibility that Spark AM exits normally and the task status is displayed as Failed. Sometimes Spark AM exits abnormally and the status is not reported in time.

Procedure

When executing a Spark Streaming task, add the Spark task parameter --conf spark.streaming.stopGracefullyOnShutdown=true and set spark.streaming.stopGracefullyOnShutdown to true.