REST API

Updated on 2022-11-18 GMT+08:00

View PDF

Function Description

Users can use the application programming interface (API) of Representational State Transfer (REST) to query more information about YARN jobs. Currently, on the REST API provided by YARN, you can only query some resources or jobs. For details of the HTTP REST API, see the following official guidelines:

http://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html

Preparing Running Environment

Install a client on the node. For example, install a client in the /opt/client directory. See details in "Software Installation > Initial Configuration > Configuring Client > Installing a Client".
Go to the /opt/client directory where the client is installed and run the following commands to initiate environment variables:
source bigdata_env

kinit component service user
NOTE:

The validity duration of kinit authentication is 24 hours. After 24 hours, you need to re-authenticate the sample with the kinit to restart the sample.
HTTPS-based access is different from HTTP-based access. When you access Yarn using HTTPS, you must ensure that the SSL protocol supported by the curl command is supported by the cluster because SSL security encryption is used. If the cluster does not support the SSL protocol, change the SSL protocol in the cluster. For example, if the Curl command only supports the TLSv1 protocol, modify the protocol configuration performing following measures:
Log in to the FusionInsight Manager portal and choose Cluster > Name of the desired cluster > Service > Yarn > Configuration > All Configurations. Type hadoop.ssl.enabled.protocols in the research box, check whether the parameter value contains TLSv1. If the parameter value does not contain TLSv1, add TLSv1 in the hadoop.ssl.enabled.protocols configuration item, and clear the value of ssl.server.exclude.cipher.list. Otherwise, Yarn cannot be accessed by using HTTPS. Then click Save Configuration and select Restart the affected services or instances. Restart the service.

NOTE:

TLSv1 has security vulnerabilities. Exercise caution when using it.

Procedure

Query the information of the jobs that run on the Yarn.

Command:
```
curl -k -i --negotiate -u : "https://10-120-85-2:8090/ws/v1/cluster/apps/"
```
NOTE:
- 10-120-85-2: host name of the active ResourceManager node.
  You can log in to FusionInsight Manager, choose Cluster > Services > Yarn > Instance, and view the Host Name of ResourceManager(Active).
- 8090: port number of the ResourceManager.
  You can log in to FusionInsight Manager, choose Cluster > Services > Yarn > Configurations > All Configurations, search for and obtain the value of yarn.resourcemanager.webapp.https.port.
If users have the admin permission, they can check jobs in some columns.
NOTE:

If the current component uses Ranger for permission control, you need to configure permission management policies based on Ranger.

The result is displayed as follows:

{
    "apps": {
        "app": [
            {
                "id": "application_1461743120947_0001",
                "user": "spark",
                "name": "Spark-JDBCServer",
                "queue": "default",
                "state": "RUNNING",
                "finalStatus": "UNDEFINED",
                "progress": 10,
                "trackingUI": "ApplicationMaster",
                "trackingUrl": "https://10-120-85-2:8090/proxy/application_1461743120947_0001/",
                "diagnostics": "AM is launched. ",
                "clusterId": 1461743120947,
                "applicationType": "SPARK",
                "applicationTags": "",
                "startedTime": 1461804906260,
                "finishedTime": 0,
                "elapsedTime": 6888848,
                "amContainerLogs": "https://10-120-85-2:8044/node/containerlogs/container_e12_1461743120947_0001_01_000001/spark",
                "amHostHttpAddress": "10-120-85-2:8044",
                "allocatedMB": 1024,
                "allocatedVCores": 1,
                "runningContainers": 1,
                "memorySeconds": 7053309,
                "vcoreSeconds": 6887,
                "preemptedResourceMB": 0,
                "preemptedResourceVCores": 0,
                "numNonAMContainerPreempted": 0,
                "numAMContainerPreempted": 0,
                "resourceRequests": [
                    {
                        "capability": {
                            "memory": 1024,
                            "virtualCores": 1
                        },
                        "nodeLabelExpression": "",
                        "numContainers": 0,
                        "priority": {
                            "priority": 0
                        },
                        "relaxLocality": true,
                        "resourceName": "*"
                    }
                ],
                "logAggregationStatus": "NOT_START",
                "amNodeLabelExpression": ""
            },
            {
                "id": "application_1461722876897_0002",
                "user": "admin",
                "name": "QuasiMonteCarlo",
                "queue": "default",
                "state": "FINISHED",
                "finalStatus": "SUCCEEDED",
                "progress": 100,
                "trackingUI": "History",
                "trackingUrl": "https://10-120-85-2:8090/proxy/application_1461722876897_0002/",
                "diagnostics": "Attempt recovered after RM restart",
                "clusterId": 1461743120947,
                "applicationType": "MAPREDUCE",
                "applicationTags": "",
                "startedTime": 1461741052993,
                "finishedTime": 1461741079483,
                "elapsedTime": 26490,
                "amContainerLogs": "https://10-120-85-2:8044/node/containerlogs/container_e11_1461722876897_0002_01_000001/admin",
                "amHostHttpAddress": "10-120-85-2:8044",
                "allocatedMB": -1,
                "allocatedVCores": -1,
                "runningContainers": -1,
                "memorySeconds": 158664,
                "vcoreSeconds": 52,
                "preemptedResourceMB": 0,
                "preemptedResourceVCores": 0,
                "numNonAMContainerPreempted": 0,
                "numAMContainerPreempted": 0,
                "amNodeLabelExpression": ""
            }
        ]
    }
}

Result analysis:

On the interface, you can query the information of jobs that are running on the YARN and obtain the common information that is displayed in Table 1.

**Table 1** Common information of jobs running on Yarn
Information	Description
user	Indicates the user who runs the job.
applicationType	Indicates the application types, such as MAPREDUCE or SPARK.
finalStatus	Indicates whether a job is executed successfully.
elapsedTime	Indicates the time to run a job.

Obtain the overall information of YARN resources.

Command:

curl -k -i --negotiate -u : "https://10-120-85-102:8090/ws/v1/cluster/metrics"

The result is displayed as follows:

{
    "clusterMetrics": {
        "appsSubmitted": 2,
        "appsCompleted": 1,
        "appsPending": 0,
        "appsRunning": 1,
        "appsFailed": 0,
        "appsKilled": 0,
        "reservedMB": 0,
        "availableMB": 23552,
        "allocatedMB": 1024,
        "reservedVirtualCores": 0,
        "availableVirtualCores": 23,
        "allocatedVirtualCores": 1,
        "containersAllocated": 1,
        "containersReserved": 0,
        "containersPending": 0,
        "totalMB": 24576,
        "totalVirtualCores": 24,
        "totalNodes": 3,
        "lostNodes": 0,
        "unhealthyNodes": 0,
        "decommissionedNodes": 0,
        "rebootedNodes": 0,
        "activeNodes": 3,
        "rmMainQueueSize": 0,
        "schedulerQueueSize": 0,
        "stateStoreQueueSize": 0
    }
}

Result analysis:

On the interface, users can query the common information of jobs that are running in the cluster, as displayed in Table 2.

**Table 2** Common information of jobs running in the cluster
Information	Description
appsSubmitted	Indicates the number of jobs that have been submitted.
appsCompleted	Indicates the number of jobs that have been completed.
appsPending	Indicates the number of jobs that have been suspended.
appsRunning	Indicates the number of jobs that are running.
appsFailed	Indicates the number of jobs that have failed.
appsKilled	Indicates the number of jobs that have been killed.
totalMB	Indicates the total memory of Yarn resources.
totalVirtualCores	Indicates the total VCores of Yarn resources.