Updated on 2022-07-11 GMT+08:00

Command

You can use YARN commands to perform operations on YARN clusters, such as starting ResourceManager, submitting applications, killing applications, querying node status, and downloading container logs.

For details about the commands, see http://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html

Common Commands

YARN commands can be used by common users and administrators. Common users can run a few of YARN commands, such as jar and logs. Administrators have permissions to run most YARN commands.

Users can run the following command to query usage of YARN:

yarn --help

Usage: Go to any directory on the YARN client, execute source command to import the environment variables, and run the command. The command format is as follows:

yarn [--config confdir] COMMAND

Table 1 describes the commands.

The version 8.1.2.2 is used as an example. Replace it with the actual version number.

Table 1 Common commands

Command

Description

resourcemanager

Runs a ResourceManager.

Notice: Before running commands, for example, the following two commands, on the server as user omm (the client must have the execute permission of user omm), export environment variables first.

  • export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_ResourceManager/etc
  • export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_ResourceManager/etc

nodemanager

Runs a NodeManager.

Notice: Before running commands, for example, the following two commands, on the server as user omm (the client must have the execute permission of user omm), export environment variables first.

  • export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_NodeManager/etc
  • export export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_NodeManager/etc

rmadmin

Runs administrator's tool for dynamically updating information.

version

Displays the version.

jar <jar>

Runs a JAR file.

logs

Obtains container logs.

classpath

Displays the class path of the Hadoop JAR package and other library files.

daemonlog

Obtains or sets the service log level.

CLASSNAME

Runs a class named by CLASSNAME.

top

Runs the cluster usage monitor tool.

-Dmapreduce.job.hdfs-servers

If OBS is connected, but the server still uses HDFS, you need to explicitly use this parameter in the command line to specify the HDFS address. The format is hdfs://{NAMESERVICE}. Replace NAMESERVICE with the HDFS NameService name.

If the current HDFS has multiple NameService instances, you need to specify all NameService instances and separate them with commas (,), for example, hdfs://nameservice1,hdfs://nameservice2.

Superior Scheduler Command

Superior Scheduler Engine provides a CLI that display detail information of Superior Scheduler Engine. For executing superior command we need to use the "<HADOOP_HOME>/bin/superior" script.

Here is the format of "superior" command.

<HADOOP_HOME>/bin/superior

Usage: superior [COMMAND | -help]
   Where COMMAND is one of:
   resourcepool                        prints resource pool status
   queue                               prints queue status
   application                         prints application status
   policy                              prints policy status

Most commands print help when invoked without parameters.

  • Superior resourcepool command :

    This command displays resourcepool and associated policy related status and configuration information.

    Superior resourcepool command can be used only by admin user and user with yarn admin rigths.

    Usage output:

    >superior resourcepool
    
    Usage: resourcepool [-help]
                        [-list]
                        [-status <resourcepoolname>]
     -help                        prints resource pool usage
     -list                        prints all resource pool summary report
     -status <resourcepoolname>   prints status and configuration of specified
                                  resource pool
    • resourcepool -list prints resource pool summary in a table format. Here is an example:
      > superior resourcepool -list
      NAME       NUMBER_MEMBER     TOTAL_RESOURCE           AVAILABLE_RESOURCE
      Pool1      4                 vcores 30,memory 1000    vcores 21,memory 80
      Pool2      100               vcores 100,memory 12800  vcores 30,memory 1000
      default    2                 vcores 64,memory 128     vcores 40,memory 28
    • resourcepool -status <resourcepoolname> prints resource pool detail information in a list format. Here is an example:
      > superior resourcepool -status default
      NAME: default
      DESCRIPTION: System generated resource pool
      TOTAL_RESOURCE: vcores 64,memory 128
      AVAILABLE_RESOURCE: vcores 40,memory 28
      NUMBER_MEMBER: 2
      MEMBERS: node1,node2
      CONFIGURATION:
      |-- RESOURCE_SELECT:
      |__RESOURCES:
  • Superior queue command

    This command displays hierarchy queue information.

    Usage output:

    >superior queue 
    
    Usage: queue [-help]
                 [-list] [-e] [[-name <queue_name>] [-r|-c]]
                 [-status <queue_name>]
     -c                     only work with -name <queue_name> option. If this
                            option is used, command  will print information of
                            specified queue and its direct children.
     -e                     only work with -list or -list -name option. If
                            this option is used, command will print effective
                            state of specified queue and all of its
                            descendants.
     -help                  prints queue sub command usage
     -list                  prints queue summary report. This option can work
                            with -name <queue_name> and -r options.
     -name <queue_name>     print specified queue, this can work with -r
                            option. By default, it will print queue's own
                            information. When -r is defined, command will
                            print all of its descendant queues. When -c is
                            defined, it will print its direct children queues.
     -r                     only work with -name <queue_name> option. If this
                            option is used, command will print information of
                            specified queue and all of its descendants.
     -status <queue_name>   prints status of specified queue
    • queue -list prints a table format of queue summary information. When command displays, command will display them based on queue hierarchy fashion. With SUBMIT ACL or ADMIN ACL rights for queue, user will be able to see queues. Here is an example:
      > superior queue -list
      NAME         STATE            NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                 RES_REQUEST
      root         OPEN|ACTIVE      10           20            100              200              vcores 100,memory 1000    vcores 200,memory 2000
      root.Q1      OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q1.Q11  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q1.Q12  CLOSE|INACTIVE   0            0             0                0                vcores 0,memory 0         vcores 0,memory 0
      root.Q2      OPEN|INACTIVE    5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q2.Q21  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
    • queue -list -name root.Q1 will display root.Q1 only.
      > superior queue -list -name root.Q1
      NAME            STATE           NRUN_APP     NPEND_APP      NRUN_CONTAINER  NPEND_REQUEST    RES_INUSE            RES_REQUEST
      root.Q1         OPEN|ACTIVE     5            10             50              100              vcores 50,memory 500 vcores 100,memory 1000
    • queue -list -name root.Q1 -r will print out root.Q1 and all of its descendents.
      > superior queue -list -name root.Q1 -r
      NAME         STATE            NRUN_APP     NPEND_APP  NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE               RES_REQUEST
      root.Q1      OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
      root.Q1.Q11  OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
      root.Q1.Q12  CLOSE|INACTIVE   0            0          0                0                vcores 0,memory 0       vcores 0,memory 0
    • queue -list -name root -c will print out root and all of its direct children
      > superior queue -list -name root -c
      NAME                  STATE             NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                RES_REQUEST
      root                  OPEN|ACTIVE       10           20            100              200              vcores 100,memory 1000   vcores 200,memory 2000
      root.Q1               OPEN|ACTIVE       5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000
      root.Q2               OPEN|INACTIVE     5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000
    • queue -status <queue_name> will display detail queue status and configuration.

      With SUBMIT ACL, user will be able to see details except ACLS of queue.

      User with ADMIN ACL rights for queue will be able to see queue details including ACL.

      > superior queue -status root.Q1
      NAME: root.Q1
      OPEN_STATE:CLOSED
      ACTIVE_STATE: INACTIVE
      EOPEN_STATE: CLOSED
      EACTIVE_STATE: INACTIVE
      LEAF_QUEUE: Yes
      NUMBER_PENDING_APPLICATION: 100
      NUMBER_RUNNING_APPLICATION: 10
      NUMBER_PENDING_REQUEST: 10
      NUMBER_RUNNING_CONTAINER: 10
      NUMBER_RESERVED_CONTAINER: 0
      RESOURCE_REQUEST: vcores 3,memory 3072
      RESOURCE_INUSE: vcores 2,memory 2048
      RESOURCE_RESERVED: vcores 0,memory 0
      CONFIGURATION:
      |-- DESCRIPTION: Spark session queue
      |-- MAX_PENDING_APPLICATION: 10000
      |--MAX_RUNNING_APPLICATION: 1000
      |--ALLOCATION_ORDER_POLICY: FIFO
      |--DEFAULT_RESOURCE_SELECT: label1
      |--MAX_MASTER_SHARE: 10%
      |--MAX_RUNNING_APPLICATION_PER_USER : -1
      |--MAX_ALLOCATION_UNIT: vcores 32,memory 12800
      |--ACL_USERS: user1,user2
      |--ACL_USERGROUPS: usergroup1,usergroup2
      |-- ACL_ADMINS: user1
      |--ACL_ADMINGROUPS: usergroup1
  • Superior application command

    This command displays application related information.

    Usage output:

    >superior application 
    
    Usage: application [-help]
                       [-list]
                       [-status <application_id>]
     -help                      prints application sub command usage
     -list                      prints all application summary report
     -status <application_id>   prints status of specified application

    With the view access rights to application user can see application related information.

    • application -list provides summary information of all application in a table format. Here is an example:
      > superior application -list
      ID                                        QUEUE             USER     NRUN_CONTAINER          NPEND_REQUEST               NRSV_CONTAINER       RES_INUSE                        RES_REQUEST                     RES_RESERVED           
      application_1482743319705_0005            root.SEQ.queueB   hbase    1                       100                         0                    vcores 1,memory 1536             vcores 2000,memory 409600                    vcores 0,memory 0
      application_1482743319705_0006            root.SEQ.queueB   hbase    0                       1                           0                    vcores 0,memory 0                vcores 1,memory 1536                     vcores 0,memory 0
    • application -status <app_id> command displays detail information of specified application. Here is an example:
      > superior application -status application_1443067302606_0609
      ID: application_1443067302606_0609
      QUEUE: root.Q1.Q11
      USER: cchen
      RESOURCE_REQUEST: vcores 3,memory 3072
      RESOURCE_INUSE: vcores 2,memory 2048
      RESOURCE_RESERVED:vcores 1, memory 1024
      NUMBER_RUNNING_CONTAINER: 2
      NUMBER_PENDING_REQUEST: 3
      NUMBER_RESERVED_CONTAINER: 1
      MASTER_CONTAINER_ID: application_1443067302606_0609_01
      MASTER_CONTAINER_RESOURCE: node1.domain.com
      BLACKLIST: node5,node8
      DEMANDS:
      |-- PRIORITY: 20
      |-- MASTER: true
      |-- CAPABILITY: vcores 2, memory 2048
      |-- COUNT: 1
      |-- RESERVED_RES : vcores 1, memory 1024
      |-- RELAXLOCALITY: true
      |-- LOCALITY: node1/1
      |-- RESOURCESELECT: label1
      |-- PENDINGREASON: "application limit reached"
      |-- ID: application_1443067302606_0609_03
      |-- RESOURCE: node1.domain.com
      |-- RESERVED_RES: vcores 1, memory 1024
      |
      |--PRIORITY: 1
      |-- MASTER: false
      |-- CAPABILITY: vcores 1,memory 1024
      |-- COUNT: 2
      |-- RESERVED_RES: vcores 0, memory 0
      |-- RELAXLOCLITY: true
      |--LOCALITY: node1/1,node2/1,rackA/2
      |-- RESOURCESELECT: label1
      |-- PENDINGREASON: "no available resource"
      CONTAINERS:
      |-- ID: application_1443067302606_0609_01
      |-- RESOURCE: node1.domain.com
      |-- CAPABILITY: vcores 1,memory 1024
      |
      |-- ID: application_1443067302606_0609_02
      |-- RESOURCE: node2.domain.com
      |-- CAPABILITY: vcores 1,memory 1024
  • Superior policy command

    This command displays policy related information.

    Superior policy command can be used only by admin user and user with yarn admin rigths.

    Usage output:

    >superior policy
    
    Usage: policy [-help]
                  [-list <resourcepoolname>] [-u] [-detail]
                  [-status <resourcepoolname>]
     -detail                      only work with -list option to show a
                                  summary information of resource pool
                                  distribution on queues, including reserve,
                                  minimum and maximum
     -help                        prints policy sub command usage
     -list <resourcepoolname>     prints a summary information of resource
                                  pool distribution on queue
     -status <resourcepoolname>   prints pool distribution policy
                                  configuration and status of specified
                                  resource pool
     -u                           only work with -list option to show a
                                  summary information of resource pool
                                  distribution on queues and also user
                                  accounts
    • policy -list <resourcepoolname> print out a summary of queue distribution information. Here is an example:
      >superior policy -list default
      NAME: default
      TOTAL_RESOURCE: vcores 16,memory 16384
      AVAILABLE_RESOURCE: vcores 16,memory 16384
      
      NAMERES_INUSERES_REQUEST
      root.defaultvcores 0,memory 0vcores 0,memory 0
      root.productionvcores 0,memory 0vcores 0,memory 0
      root.production.BU1vcores 0,memory 0vcores 0,memory 0
      root.production.BU2 vcores 0,memory 0vcores 0,memory 0
    • policy -list <resourcepoolname> -u prints out also user level summary information
      > superior policy -list default -u
      NAME: default
      TOTAL_RESOURCE: vcores 16,memory 16384
      AVAILABLE_RESOURCE: vcores 16,memory 16384
      
      NAMERES_INUSERES_REQUEST
      root.defaultvcores 0,memory 0vcores 0,memory 0
      root.default.[_others_]vcores 0,memory 0vcores 0,memory 0
      root.productionvcores 0,memory 0vcores 0,memory 0
      root.production.BU1vcores 0,memory 0vcores 0,memory 0
      root.production.BU1.[_others_]vcores 0,memory 0vcores 0,memory 0
      root.production.BU2vcores 0,memory 0vcores 0,memory 0
      root.production.BU2.[_others_]vcores 0,memory 0vcores 0,memory 0
    • policy -status <resourcepoolname> print out policy detail of specified resource pool. Here is an example:
      > superior policy -status pool1
      NAME: pool1
      TOTAL_RESOURCE: vcores 64,memory 128
      AVAILABLE_RESOURCE: vcores 40,memory 28
      QUEUES:
      |-- NAME: root.Q1
      |-- RESOURCE_USE: vcores 20, memory 1000
      |-- RESOURCE_REQUEST: vcores 2,memory 100
      |--RESERVE: vcores 10, memory 4096
      |--MINIMUM: vcore 11, memory 4096
      |--MAXIMUM: vcores 500, memory 100000
      |--CONFIGURATION:
      |-- SHARE: 50%
      |-- RESERVE: vcores 10, memory 4096
      |-- MINIMUM: vcores 11, memory 4096
      |-- MAXIMUM: vcores 500, memory 100000
      |-- QUEUES:
      |-- NAME: root.Q1.Q11
      |-- RESOURCE_USE: vcores 15, memory, 500
      |-- RESOURCE_REQUEST: vcores 1, memory 50
      |-- RESERVE: vcores 0, memory 0
      |-- MINIMUM: vcores 0, memory 0
      |-- MAXIMUM: vcores -1, memory -1
      |-- USER_ACCOUNTS:
      |-- NAME: user1
      |-- RESOURCE_USE: vcores 1, memory 10
      |-- RESOURCE_REQUEST: vcores 1, memory 50
      |
      |-- NAME: OTHERS
      |--RESOURCE_USE: vcores 0, memory 0
      |- RESOURCE_REQUEST: vcores 0, memory 0
      |-- CONFIGURATION:
      |-- SHARE: 100%
      |-- USER_POLICY:
      |-- NAME: user1
      |-- WEIGHT: 10
      |
      |-- NAME: OTHERS
      |-- WEIGHT: 1
      |-- MAXIMUM: vcores 10, memory 1000