Why Does the Cluster Port Fail to Connect When a Client Outside the Cluster Is Installed or Used?

Question

When a client outside the cluster is installed or used, the Spark task port sometimes fails to be connected.

Exception information: Failed to bind SparkUi

Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.

Causes

• The network between the cluster node and the client node is disconnected.
• The firewall on the client node is not disabled.
• If the port is occupied, each Spark task occupies a SparkUI port. The default port number is 22600. If the port is occupied, increase the port number in sequence and try again. However, there are only 16 retries by default. After the 16 retries, this task is aborted.
• The Spark configuration parameters on the client are incorrect.
• The code is incorrect.

Solution

The application cannot access the IP address and port number of SparkUI. The possible causes are as follows:

• Check whether the cluster node communicates with the client node.

  Run the following command on the client node to check whether the cluster node mapping is configured in the /etc/hosts file on the client node:

  ping SparkUI IP address

  If the IP address cannot be pinged, check the mapping configuration and network configuration.

• Disable the firewall on the client node.

  Run the following command to check whether the function is disabled:

  systemctl status firewalld (The query command varies depending on the OS. This command uses CentOS as an example.)

  As shown in the following figure, dead indicates that the function is disabled.

  

  If the firewall is enabled, the communication is affected. Run the following command to disable the firewall:

  service firewalld stop (The query command varies depending on the OS. This command uses CentOS as an example.)

• Check whether the port is occupied.

  ssh -v -p port username@ip

  If the message "Connection established" is displayed, it indicates that the connection is successful and the port is occupied.

  

  The Spark UI port range is determined by the spark.random.port.min and spark.random.port.max parameters in the spark-defaults.conf configuration file. If all ports in the range are used,

  No port is available and the connection fails.

  Solution: Set spark.port.maxRetries to 50 and adjust the random port range of the executor to spark.random.port.max plus 100.

• View Spark configuration parameters:

  Run the cat spark-env.sh command on the client node to check whether the SPARK_LOCAL_HOSTNAME value is the IP address of the local host.

  This problem may occur when the client is directly copied from another node and the configuration parameters are not modified.

  Change the value of SPARK_LOCAL_HOSTNAME to the IP address of the local host.

  Note: If the cluster uses EIPs for communication, you need to add the following configuration:

  1. Add spark.driver.host=Elastic IP address of the client node to spark-default.conf.
  2. Add spark.driver.bindAddress=IP address of the local host to spark-default.conf.
  3. Add SPARK_LOCAL_HOSTNAME=Elastic IP address of the client node to spark-env.sh.
• If the communication and configuration are normal, check the code.

  When Spark starts a task, sparkDriverEnv is created on the client and bound to DRIVER_BIND_ADDRESS. This logic does not go to the server. So, this problem occurs because sparkDriver cannot obtain the corresponding host IP address due to abnormal OS environment of the client node.

  You can run the export SPARK_LOCAL_HOSTNAME=172.0.0.1 command or set spark.driver.bindAddress to 127.0.0.1 so that the driver that submits tasks can load loopbackAddress.