Why the Name of the Spark Application Submitted in Yarn-Cluster Mode Does not Take Effect?

Question

The name of the Spark application submitted in yarn-cluster mode does not take effect, whereas the Spark application name submitted in yarn-client mode takes effect. As shown in Figure 1, the first application is submitted in yarn-client mode and the application name Spark Pi takes effect. However, the setAppName execution sequence of a task submitted in yarn-client mode is different from that submitted in yarn-cluster mode.

Figure 1 Submitting the application

Answer

The reason is that the setAppName execution sequence of a task submitted in yarn-client mode is different from that submitted in yarn-cluster mode. In yarn-client mode, the setAppName is read before the application is registered in yarn. However, in yarn-cluster mode, the setAppName is read after the application registers with yarn, so the name of the second application does not take effect.

Troubleshooting solution

When submitting tasks using the spark-submit script, set --name the same as the application name in sparkconf.setAppName (appname).

For example, if the application name is Spark Pi, in sparkconf.setAppName (appname) in yarn-cluster mode, run the following command:

./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --name SparkPi lib/spark-examples*.jar 10