# Yarn-client模式提交Spark任务时ApplicationMaster尝试启动两次失败
#### 问题现象
Yarn-client模式提交任务AppMaster尝试启动两次失败。
#### 原因分析
1. Driver端异常:
   ```
   16/05/11 18:10:56 INFO Client: 
   client token: N/A
   diagnostics: Application application_1462441251516_0024 failed 2 times due to AM Container for appattempt_1462441251516_0024_000002 exited with  exitCode: 10
   For more detailed output, check the application tracking page:https://hdnode5:26001/cluster/app/application_1462441251516_0024 Then click on links to logs of each attempt.
   Diagnostics: Exception from container-launch.
   Container id: container_1462441251516_0024_02_000001
   ```
   
2. 在ApplicationMaster日志中，异常如下:
   ```
   2016-05-12 10:21:23,715 | ERROR | [main] | Failed to connect to driver at 192.168.30.57:23867, retrying ... | org.apache.spark.Logging$class.logError(Logging.scala:75)
   2016-05-12 10:21:24,817 | ERROR | [main] | Failed to connect to driver at 192.168.30.57:23867, retrying ... | org.apache.spark.Logging$class.logError(Logging.scala:75)
   2016-05-12 10:21:24,918 | ERROR | [main] | Uncaught exception:  | org.apache.spark.Logging$class.logError(Logging.scala:96)
   org.apache.spark.SparkException: Failed to connect to driver!
   at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:426)
   at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:292)
   …
   2016-05-12 10:21:24,925 | INFO  | [Thread-1] | Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!) | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
   ```
   Spark-client模式任务Driver运行在客户端节点上(通常是集群外的某个节点)，启动时先在集群中启动AppMaster进程，进程启动后要向Driver进程注册信息，注册成功后，任务才能继续。从AppMaster日志中可以看出，无法连接至Driver，所以任务失败。
   
 
#### 解决办法
1. 请检查Driver进程所在的IP是否可以**ping**通。
2. 启动一个Spark PI任务，会有类似如下打印信息。
   
   ```
   16/05/11 18:07:20 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.100:23662]
   16/05/11 18:07:20 INFO Utils: Successfully started service 'sparkDriver' on port 23662.
   ```
   
   
3. 在该节点，也就是[2]中示例的192.168.1.100上执行**netstat - anp \| grep 23662** 看下此端口是否打开，如下打印信息表明，相关端口是打开的。
   
   ```
   tcp        0      0  ip:port    :::*                    LISTEN      107274/java        
   tcp        0      0  ip:port   ip:port                  ESTABLISHED 107274/java
   ```
   
   
4. 在AppMaster启动的节点执行**telnet 192.168.1.100 23662** 看下是否可以连通该端口，请使用**root** 用户和**omm** 用户都执行一遍。 如果出现**Escape character is '\^\]'** 类似打印则说明可以连通，如果出现**connection refused** 则表示失败，无法连接到相关端口。
   
   如果相关端口打开，但是从别的节点无法连通到该端口，则需要排查下相关网络配置。
   ![](https://support.huaweicloud.com/trouble-mrs/public_sys-resources/note_3.0-zh-cn.png)
   23662这个端口每次都是随机的，所以要根据自己启动任务打开的端口来测试。