REST API

Function Description

You can use HTTP REST APIs to query more information about Yarn jobs. Currently, you can only query some resources or jobs using REST APIs provided by Yarn. For details of HTTP REST APIs, see the following official guidelines:

http://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html

Preparing Running Environment

1. Install a client on the node. For example, install a client in the /opt/clientdirectory. See details in "Installing a Client."
2. Go to the /opt/clientdirectory where the client is installed and run the following commands to initiate environment variables:

   sourcebigdata_env

Procedure

1. Query the information about jobs that run on the Yarn.
   • Command:

     curl -k -i --negotiate -u : "http://10-120-85-2:8088/ws/v1/cluster/apps/"

     In the commands, 10-120-85-2 is the hostname of the main node ResourceManager and 8088 is the port number of ResourceManager.

   • The running result is displayed as follows:

     {
         "apps": {
             "app": [
                 {
                     "id": "application_1461743120947_0001",
                     "user": "spark",
                     "name": "Spark-JDBCServer",
                     "queue": "default",
                     "state": "RUNNING",
                     "finalStatus": "UNDEFINED",
                     "progress": 10,
                     "trackingUI": "ApplicationMaster",
                     "trackingUrl": "http://10-120-85-2:8088/proxy/application_1461743120947_0001/",
                     "diagnostics": "AM is launched. ",
                     "clusterId": 1461743120947,
                     "applicationType": "SPARK",
                     "applicationTags": "",
                     "startedTime": 1461804906260,
                     "finishedTime": 0,
                     "elapsedTime": 6888848,
                     "amContainerLogs": "https://10-120-85-2:8088/node/containerlogs/container_e12_1461743120947_0001_01_000001/spark",
                     "amHostHttpAddress": "10-120-85-2:8088",
                     "allocatedMB": 1024,
                     "allocatedVCores": 1,
                     "runningContainers": 1,
                     "memorySeconds": 7053309,
                     "vcoreSeconds": 6887,
                     "preemptedResourceMB": 0,
                     "preemptedResourceVCores": 0,
                     "numNonAMContainerPreempted": 0,
                     "numAMContainerPreempted": 0,
                     "resourceRequests": [
                         {
                             "capability": {
                                 "memory": 1024,
                                 "virtualCores": 1
                             },
                             "nodeLabelExpression": "",
                             "numContainers": 0,
                             "priority": {
                                 "priority": 0
                             },
                             "relaxLocality": true,
                             "resourceName": "*"
                         }
                     ],
                     "logAggregationStatus": "NOT_START",
                     "amNodeLabelExpression": ""
                 },
                 {
                     "id": "application_1461722876897_0002",
                     "user": "admin",
                     "name": "QuasiMonteCarlo",
                     "queue": "default",
                     "state": "FINISHED",
                     "finalStatus": "SUCCEEDED",
                     "progress": 100,
                     "trackingUI": "History",
                     "trackingUrl": "http://10-120-85-2:8088/proxy/application_1461722876897_0002/",
                     "diagnostics": "Attempt recovered after RM restart",
                     "clusterId": 1461743120947,
                     "applicationType": "MAPREDUCE",
                     "applicationTags": "",
                     "startedTime": 1461741052993,
                     "finishedTime": 1461741079483,
                     "elapsedTime": 26490,
                     "amContainerLogs": "http://10-120-85-2:8088/node/containerlogs/container_e11_1461722876897_0002_01_000001/admin",
                     "amHostHttpAddress": "10-120-85-2:8088",
                     "allocatedMB": -1,
                     "allocatedVCores": -1,
                     "runningContainers": -1,
                     "memorySeconds": 158664,
                     "vcoreSeconds": 52,
                     "preemptedResourceMB": 0,
                     "preemptedResourceVCores": 0,
                     "numNonAMContainerPreempted": 0,
                     "numAMContainerPreempted": 0,
                     "amNodeLabelExpression": ""
                 }
             ]
         }
     }

   • Result analysis:
     On the interface, you can query the information of jobs that are running on the Yarn and obtain the common information that is displayed in Table 1.

     Table 1 Common information of jobs running on Yarn

     Information	Description
     user	Indicates the user who runs the job.
     applicationType	Indicates the application types, such as MAPREDUCE or SPARK.
     finalStatus	Indicates whether a job is executed successfully.
     elapsedTime	Indicates the time to run a job.

2. Obtain the overall information of Yarn resources.
   • Command:

     curl -k -i --negotiate -u : "http://10-120-85-102:8088/ws/v1/cluster/metrics"

   • The running result is displayed as follows:

     {
         "clusterMetrics": {
             "appsSubmitted": 2,
             "appsCompleted": 1,
             "appsPending": 0,
             "appsRunning": 1,
             "appsFailed": 0,
             "appsKilled": 0,
             "reservedMB": 0,
             "availableMB": 23552,
             "allocatedMB": 1024,
             "reservedVirtualCores": 0,
             "availableVirtualCores": 23,
             "allocatedVirtualCores": 1,
             "containersAllocated": 1,
             "containersReserved": 0,
             "containersPending": 0,
             "totalMB": 24576,
             "totalVirtualCores": 24,
             "totalNodes": 3,
             "lostNodes": 0,
             "unhealthyNodes": 0,
             "decommissionedNodes": 0,
             "rebootedNodes": 0,
             "activeNodes": 3,
             "rmMainQueueSize": 0,
             "schedulerQueueSize": 0,
             "stateStoreQueueSize": 0
         }
     }

   • Result analysis:
     On the interface, users can query the common information of jobs that are running in the cluster, as displayed in Table 2.

     Table 2 Common information of jobs running in the cluster

     Information	Description
     appsSubmitted	Indicates the number of jobs that have been submitted.
     appsCompleted	Indicates the number of jobs that have been completed.
     appsPending	Indicates the number of jobs that have been suspended.
     appsRunning	Indicates the number of jobs that are running.
     appsFailed	Indicates the number of jobs that have failed.
     appsKilled	Indicates the number of jobs that have been killed.
     totalMB	Indicates the total memory of Yarn resources.
     totalVirtualCores	Indicates the total VCores of Yarn resources.