Why the Name of the Spark Application Submitted in Yarn-Cluster Mode Does not Take Effect?

Question

The name of the Spark application submitted in yarn-cluster mode does not take effect, whereas the Spark application name submitted in yarn-client mode takes effect. As shown in Figure 1, the first application is submitted in yarn-client mode and the application name Spark Pi takes effect. However, the setAppName execution sequence of a task submitted in yarn-client mode is different from that submitted in yarn-cluster mode.

Figure 1 Submitting the application

Answer

The reason is that the setAppName execution sequence of a task submitted in yarn-client mode is different from that submitted in yarn-cluster mode. In yarn-client mode, the setAppName is read before the application is registered in yarn. However, in yarn-cluster mode, the setAppName is read after the application registers with yarn, so the name of the second application does not take effect.

Solution

When submitting tasks using the spark-submit script, set --name the same as the application name in sparkconf.setAppName (appname).

For example, if the application name is Spark Pi, run the following command to add the application name after --name when submitting the application in yarn-cluster mode:

./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --name SparkPi jars/original-spark-examples*.jar 10