YARN REST API接口介绍

功能简介

通过HTTP REST API来查看更多Yarn任务的信息。目前Yarn的REST接口只能进行一些资源或者任务的查询。完整和详细的接口请直接参考官网上的描述以了解其使用：http://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html

准备运行环境

1. 在节点上安装客户端，例如安装到“/opt/client”目录。
2. 进入客户端安装目录“/opt/client”，执行下列命令初始化环境变量。

   source bigdata_env

操作步骤

1. 获取运行在Yarn上的任务的具体信息。
   • 命令：

     curl -k -i --negotiate -u : "http://10-120-85-2:8088/ws/v1/cluster/apps/"

     其中10-120-85-2为ResourceManager主节点的hostname，8088为ResourceManager的端口号。

   • 运行结果：

     {
         "apps": {
             "app": [
                 {
                     "id": "application_1461743120947_0001",
                     "user": "spark",
                     "name": "Spark-JDBCServer",
                     "queue": "default",
                     "state": "RUNNING",
                     "finalStatus": "UNDEFINED",
                     "progress": 10,
                     "trackingUI": "ApplicationMaster",
                     "trackingUrl": "http://10-120-85-2:8088/proxy/application_1461743120947_0001/",
                     "diagnostics": "AM is launched. ",
                     "clusterId": 1461743120947,
                     "applicationType": "SPARK",
                     "applicationTags": "",
                     "startedTime": 1461804906260,
                     "finishedTime": 0,
                     "elapsedTime": 6888848,
                     "amContainerLogs": "http://10-120-85-2:8088/node/containerlogs/container_e12_1461743120947_0001_01_000001/spark",
                     "amHostHttpAddress": "10-120-85-2:8088",
                     "allocatedMB": 1024,
                     "allocatedVCores": 1,
                     "runningContainers": 1,
                     "memorySeconds": 7053309,
                     "vcoreSeconds": 6887,
                     "preemptedResourceMB": 0,
                     "preemptedResourceVCores": 0,
                     "numNonAMContainerPreempted": 0,
                     "numAMContainerPreempted": 0,
                     "resourceRequests": [
                         {
                             "capability": {
                                 "memory": 1024,
                                 "virtualCores": 1
                             },
                             "nodeLabelExpression": "",
                             "numContainers": 0,
                             "priority": {
                                 "priority": 0
                             },
                             "relaxLocality": true,
                             "resourceName": "*"
                         }
                     ],
                     "logAggregationStatus": "NOT_START",
                     "amNodeLabelExpression": ""
                 },
                 {
                     "id": "application_1461722876897_0002",
                     "user": "admin",
                     "name": "QuasiMonteCarlo",
                     "queue": "default",
                     "state": "FINISHED",
                     "finalStatus": "SUCCEEDED",
                     "progress": 100,
                     "trackingUI": "History",
                     "trackingUrl": "http://10-120-85-2:8088/proxy/application_1461722876897_0002/",
                     "diagnostics": "Attempt recovered after RM restart",
                     "clusterId": 1461743120947,
                     "applicationType": "MAPREDUCE",
                     "applicationTags": "",
                     "startedTime": 1461741052993,
                     "finishedTime": 1461741079483,
                     "elapsedTime": 26490,
                     "amContainerLogs": "http://10-120-85-2:8088/node/containerlogs/container_e11_1461722876897_0002_01_000001/admin",
                     "amHostHttpAddress": "10-120-85-2:8088",
                     "allocatedMB": -1,
                     "allocatedVCores": -1,
                     "runningContainers": -1,
                     "memorySeconds": 158664,
                     "vcoreSeconds": 52,
                     "preemptedResourceMB": 0,
                     "preemptedResourceVCores": 0,
                     "numNonAMContainerPreempted": 0,
                     "numAMContainerPreempted": 0,
                     "amNodeLabelExpression": ""
                 }
             ]
         }
     }

   • 结果分析：
     通过这个接口，可以查询当前集群中Yarn上的任务，并且可以得到如下表1。

     表1 常用信息

     参数	参数描述
     user	运行这个任务的用户。
     applicationType	例如MAPREDUCE或者SPARK等。
     finalStatus	可以知道任务是成功还是失败。
     elapsedTime	任务运行的时间。

2. 获取Yarn资源的总体信息
   • 命令：

     curl -k -i --negotiate -u : "http://10-120-85-102:8088/ws/v1/cluster/metrics"

   • 运行结果：

     {
         "clusterMetrics": {
             "appsSubmitted": 2,
             "appsCompleted": 1,
             "appsPending": 0,
             "appsRunning": 1,
             "appsFailed": 0,
             "appsKilled": 0,
             "reservedMB": 0,
             "availableMB": 23552,
             "allocatedMB": 1024,
             "reservedVirtualCores": 0,
             "availableVirtualCores": 23,
             "allocatedVirtualCores": 1,
             "containersAllocated": 1,
             "containersReserved": 0,
             "containersPending": 0,
             "totalMB": 24576,
             "totalVirtualCores": 24,
             "totalNodes": 3,
             "lostNodes": 0,
             "unhealthyNodes": 0,
             "decommissionedNodes": 0,
             "rebootedNodes": 0,
             "activeNodes": 3,
             "rmMainQueueSize": 0,
             "schedulerQueueSize": 0,
             "stateStoreQueueSize": 0
         }
     }

   • 结果分析：
     通过这个接口，可以查询当前集群中如表2。

     表2 常用信息

     参数	参数描述
     appsSubmitted	已经提交的任务数。
     appsCompleted	已经完成的任务数。
     appsPending	正在挂起的任务数。
     appsRunning	正在运行的任务数。
     appsFailed	已经失败的任务数。
     appsKilled	已经被kill的任务数。
     totalMB	Yarn资源总的内存。
     totalVirtualCores	Yarn资源总的VCore数。