Failed to Connect to the Driver When a Node Outside the Cluster Submits a Spark Job to Yarn

Issue

When a node outside the cluster uses the client mode to submit a Spark task to Yarn, the task fails and an error message is displayed, indicating that the driver cannot be connected.

Symptom

Nodes outside the cluster can communicate with each node in the cluster. When a node outside the cluster submits a Spark task to Yarn in client mode, the task fails and an error message is displayed, indicating that the driver cannot be connected.

Cause Analysis

When a Spark task is submitted in the client mode, the driver process of Spark is on the client side, and the executor needs to interact with the driver to run the job.

If the NodeManager fails to connect to the node where the client is located, the following error is reported:

Procedure

Specify the IP address of the driver in the Spark configuration of the client.

Add spark.driver.host=driverIP to <Client installation path>/Spark/spark/conf/spark-defaults.conf and run the Spark task again.

Summary and Suggestions

You are advised to submit jobs in cluster mode.