Failed to Connect to the Driver When a Spark Job Is Submitted on a Node Outside the Cluster

Symptom

Nodes outside the cluster can communicate with each node in the cluster. When a node outside the cluster submits a Spark task to Yarn in client mode, the submission fails and an error message is displayed, indicating that the Driver cannot be connected.

Cause Analysis

When a Spark task is submitted in the client mode, the driver process of Spark is on the client side, and the executor needs to interact with the driver to run the job.

If the NodeManager fails to connect to the node where the client is located, the following error is reported:

Procedure

Specify the IP address of the driver in the Spark configuration of the client.

Add spark.driver.host=driverIP to <Client installation path>/Spark/spark/conf/spark-defaults.conf and run the Spark task again.

Summary and Suggestions

You are advised to submit jobs in cluster mode.