Why Does the Cluster Port Fail to Connect When a Client Outside the Cluster Is Installed or Used?

Question

When a client outside the cluster is installed or used, the Spark task port sometimes fails to be connected.

Exception information: "Failed to bind SparkUi"

Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.

Solutions

The application cannot access the IP address and port of Spark UI. The possible causes are as follows:

• Possible cause 1: The network between the cluster node and the client node is disconnected.

  Solution:

  Run the following command on the client node to check whether the cluster node mapping is configured in the /etc/hosts file on the client node:

  ping SparkUI IP address

  If the IP address cannot be pinged, check the mapping configuration and network configuration.

• Possible cause 2: The firewall on the client node is enabled.

  Solution:

  Run the following command to check whether the firewall is disabled:

  systemctl status firewalld (The query command varies depending on the OS. This command uses CentOS as an example.)

  As shown in the following figure, dead indicates that the firewall is disabled.

  

  If the firewall is enabled, the communication is affected. Run the following command to disable the firewall:

  service firewalld stop (The query command varies depending on the OS. This command uses CentOS as an example.)

• Possible cause 3: If the port is already in use, each Spark will occupy a new Spark UI port. The default port is 22600. If that is taken, increase the port number in sequence and try again. However, there are only 16 retries by default. After the 16 retries, this task is aborted.

  Check whether the port is in use.

  ssh -v -p port username@ip

  If the message "Connection established" appears, it means that the connection was successful and the port is in use.

  

  The Spark UI port range is determined by the spark.random.port.min and spark.random.port.max parameters in the spark-defaults.conf configuration file. If all ports in the range are used, no port is available and the connection fails.

  Solution: Set spark.port.maxRetries to 50 and adjust the random port range of the executor to spark.random.port.max plus 100.

• Possible cause 4: The Spark configuration on the client is incorrect.

  Solution:

  Run the cat spark-env.sh command on the client node to check whether the SPARK_LOCAL_HOSTNAME value is the IP address of the local host.

  This problem may occur when the client is directly copied from another node and the configuration parameters are not modified.

  Change the value of SPARK_LOCAL_HOSTNAME to the IP address of the local host.

  Note: If the cluster uses EIPs for communication, you need to add the following configuration:

  1. Add spark.driver.host=EIP of the client node to spark-default.conf.
  2. Add spark.driver.bindAddress=IP address of the local host to spark-default.conf.
  3. Add SPARK_LOCAL_HOSTNAME=EIP of the client node to spark-env.sh.
• Possible cause 5: code error

  Solution:

  When Spark starts a task, sparkDriverEnv is created on the client and bound to DRIVER_BIND_ADDRESS. This does not work on the server. So, this problem occurs because sparkDriver cannot obtain the host IP address due to abnormal OS environment of the client node.

  You can set spark.driver.bindAddress=127.0.0.1 to load the loopback address to the driver that submits tasks.