Failed to Connect to ResourceManager When a Spark Task Is Submitted

Symptom

The connection to ResourceManager is abnormal. As a result, Spark tasks fail to be submitted.

Cause Analysis

1. The following error information is displayed on the Driver, indicating that port 26004 connecting to the active and standby ResourceManager nodes is rejected:

   15/08/19 18:36:16 INFO RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over 33 after 1 fail over attempts. Trying to fail over after sleeping for 17448ms. 
    java.net.ConnectException: Call From ip0 to ip1:26004 failed on connection exception: java.net.ConnectException: Connection refused.
   INFO RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over 32 after 2 fail over attempts. Trying to fail over after sleeping for 16233ms. 
    java.net.ConnectException: Call From ip0 to ip2:26004 failed on connection exception: java.net.ConnectException: Connection refused;

2. On MRS Manager, check whether ResourceManager is running properly, as shown in Figure 1. If Yarn is faulty or an unknown exception occurs on a Yarn service instance, ResourceManager of the cluster may be abnormal.
   Figure 1 Service status
   
3. Check whether the client is the latest one in the cluster.

   Check whether the ResourceManager instance has been migrated in the cluster. (Uninstall a ResourceManager instance and add it back to other nodes.)

4. On MRS Manager, click Audit to view audit logs and check whether related operations are recorded.

   Run the ping command to check whether the IP address can be pinged.

Solution

• If ResourceManager is abnormal, see the Yarn-related sections to rectify the fault.
• If the client is not the latest, download the client again.
• If the IP address cannot be pinged, contact network management personnel to check the network.