Command

Common Commands

YARN commands can be used by common users and administrators. Common users can run a few of YARN commands, such as jar and logs. Administrators have permissions to run most YARN commands.

Users can run the following command to query usage of YARN:

yarn --help

Usage: Go to any directory on the YARN client, execute source command to import the environment variables, and run the command. The command format is as follows:

yarn [--config confdir] COMMAND

Table 1 describes the commands.

The version 8.1.2.2 is used as an example. Replace it with the actual version number.

Table 1 Common commands

Command	Description
resourcemanager	Runs a ResourceManager. Notice: Before running commands, for example, the following two commands, on the server as user omm (the client must have the execute permission of user omm), export environment variables first. export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_ResourceManager/etc export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_ResourceManager/etc
nodemanager	Runs a NodeManager. Notice: Before running commands, for example, the following two commands, on the server as user omm (the client must have the execute permission of user omm), export environment variables first. export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_NodeManager/etc export export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_8.1.2.2/1_10_NodeManager/etc
rmadmin	Runs administrator's tool for dynamically updating information.
version	Displays the version.
jar <jar>	Runs a JAR file.
logs	Obtains container logs.
classpath	Displays the class path of the Hadoop JAR package and other library files.
daemonlog	Obtains or sets the service log level.
CLASSNAME	Runs a class named by CLASSNAME.
top	Runs the cluster usage monitor tool.
-Dmapreduce.job.hdfs-servers	If OBS is connected, but the server still uses HDFS, you need to explicitly use this parameter in the command line to specify the HDFS address. The format is hdfs://{NAMESERVICE}. Replace NAMESERVICE with the HDFS NameService name. If the current HDFS has multiple NameService instances, you need to specify all NameService instances and separate them with commas (,), for example, hdfs://nameservice1,hdfs://nameservice2.

Superior Scheduler Command

Superior Scheduler Engine provides a CLI that display detail information of Superior Scheduler Engine. For executing superior command we need to use the "<HADOOP_HOME>/bin/superior" script.

Here is the format of "superior" command.

<HADOOP_HOME>/bin/superior

Usage: superior [COMMAND | -help]
   Where COMMAND is one of:
   resourcepool                        prints resource pool status
   queue                               prints queue status
   application                         prints application status
   policy                              prints policy status

Most commands print help when invoked without parameters.

• Superior resourcepool command :

  This command displays resourcepool and associated policy related status and configuration information.

  

  Superior resourcepool command can be used only by admin user and user with yarn admin rigths.

  Usage output:

  >superior resourcepool
  
  Usage: resourcepool [-help]
                      [-list]
                      [-status <resourcepoolname>]
   -help                        prints resource pool usage
   -list                        prints all resource pool summary report
   -status <resourcepoolname>   prints status and configuration of specified
                                resource pool

  • resourcepool -list prints resource pool summary in a table format. Here is an example:

    > superior resourcepool -list
    NAME       NUMBER_MEMBER     TOTAL_RESOURCE           AVAILABLE_RESOURCE
    Pool1      4                 vcores 30,memory 1000    vcores 21,memory 80
    Pool2      100               vcores 100,memory 12800  vcores 30,memory 1000
    default    2                 vcores 64,memory 128     vcores 40,memory 28

  • resourcepool -status <resourcepoolname> prints resource pool detail information in a list format. Here is an example:

    > superior resourcepool -status default
    NAME: default
    DESCRIPTION: System generated resource pool
    TOTAL_RESOURCE: vcores 64,memory 128
    AVAILABLE_RESOURCE: vcores 40,memory 28
    NUMBER_MEMBER: 2
    MEMBERS: node1,node2
    CONFIGURATION:
    |-- RESOURCE_SELECT:
    |__RESOURCES:

• Superior queue command

  This command displays hierarchy queue information.

  Usage output:

  >superior queue 
  
  Usage: queue [-help]
               [-list] [-e] [[-name <queue_name>] [-r|-c]]
               [-status <queue_name>]
   -c                     only work with -name <queue_name> option. If this
                          option is used, command  will print information of
                          specified queue and its direct children.
   -e                     only work with -list or -list -name option. If
                          this option is used, command will print effective
                          state of specified queue and all of its
                          descendants.
   -help                  prints queue sub command usage
   -list                  prints queue summary report. This option can work
                          with -name <queue_name> and -r options.
   -name <queue_name>     print specified queue, this can work with -r
                          option. By default, it will print queue's own
                          information. When -r is defined, command will
                          print all of its descendant queues. When -c is
                          defined, it will print its direct children queues.
   -r                     only work with -name <queue_name> option. If this
                          option is used, command will print information of
                          specified queue and all of its descendants.
   -status <queue_name>   prints status of specified queue

  • queue -list prints a table format of queue summary information. When command displays, command will display them based on queue hierarchy fashion. With SUBMIT ACL or ADMIN ACL rights for queue, user will be able to see queues. Here is an example:

    > superior queue -list
    NAME         STATE            NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                 RES_REQUEST
    root         OPEN|ACTIVE      10           20            100              200              vcores 100,memory 1000    vcores 200,memory 2000
    root.Q1      OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
    root.Q1.Q11  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
    root.Q1.Q12  CLOSE|INACTIVE   0            0             0                0                vcores 0,memory 0         vcores 0,memory 0
    root.Q2      OPEN|INACTIVE    5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
    root.Q2.Q21  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000

  • queue -list -name root.Q1 will display root.Q1 only.

    > superior queue -list -name root.Q1
    NAME            STATE           NRUN_APP     NPEND_APP      NRUN_CONTAINER  NPEND_REQUEST    RES_INUSE            RES_REQUEST
    root.Q1         OPEN|ACTIVE     5            10             50              100              vcores 50,memory 500 vcores 100,memory 1000

  • queue -list -name root.Q1 -r will print out root.Q1 and all of its descendents.

    > superior queue -list -name root.Q1 -r
    NAME         STATE            NRUN_APP     NPEND_APP  NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE               RES_REQUEST
    root.Q1      OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
    root.Q1.Q11  OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
    root.Q1.Q12  CLOSE|INACTIVE   0            0          0                0                vcores 0,memory 0       vcores 0,memory 0

  • queue -list -name root -c will print out root and all of its direct children

    > superior queue -list -name root -c
    NAME                  STATE             NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                RES_REQUEST
    root                  OPEN|ACTIVE       10           20            100              200              vcores 100,memory 1000   vcores 200,memory 2000
    root.Q1               OPEN|ACTIVE       5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000
    root.Q2               OPEN|INACTIVE     5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000

  • queue -status <queue_name> will display detail queue status and configuration.

    With SUBMIT ACL, user will be able to see details except ACLS of queue.

    User with ADMIN ACL rights for queue will be able to see queue details including ACL.

    > superior queue -status root.Q1
    NAME: root.Q1
    OPEN_STATE:CLOSED
    ACTIVE_STATE: INACTIVE
    EOPEN_STATE: CLOSED
    EACTIVE_STATE: INACTIVE
    LEAF_QUEUE: Yes
    NUMBER_PENDING_APPLICATION: 100
    NUMBER_RUNNING_APPLICATION: 10
    NUMBER_PENDING_REQUEST: 10
    NUMBER_RUNNING_CONTAINER: 10
    NUMBER_RESERVED_CONTAINER: 0
    RESOURCE_REQUEST: vcores 3,memory 3072
    RESOURCE_INUSE: vcores 2,memory 2048
    RESOURCE_RESERVED: vcores 0,memory 0
    CONFIGURATION:
    |-- DESCRIPTION: Spark session queue
    |-- MAX_PENDING_APPLICATION: 10000
    |--MAX_RUNNING_APPLICATION: 1000
    |--ALLOCATION_ORDER_POLICY: FIFO
    |--DEFAULT_RESOURCE_SELECT: label1
    |--MAX_MASTER_SHARE: 10%
    |--MAX_RUNNING_APPLICATION_PER_USER : -1
    |--MAX_ALLOCATION_UNIT: vcores 32,memory 12800
    |--ACL_USERS: user1,user2
    |--ACL_USERGROUPS: usergroup1,usergroup2
    |-- ACL_ADMINS: user1
    |--ACL_ADMINGROUPS: usergroup1

• Superior application command

  This command displays application related information.

  Usage output:

  >superior application 
  
  Usage: application [-help]
                     [-list]
                     [-status <application_id>]
   -help                      prints application sub command usage
   -list                      prints all application summary report
   -status <application_id>   prints status of specified application

  With the view access rights to application user can see application related information.

  • application -list provides summary information of all application in a table format. Here is an example:

    > superior application -list
    ID                                        QUEUE             USER     NRUN_CONTAINER          NPEND_REQUEST               NRSV_CONTAINER       RES_INUSE                        RES_REQUEST                     RES_RESERVED           
    application_1482743319705_0005            root.SEQ.queueB   hbase    1                       100                         0                    vcores 1,memory 1536             vcores 2000,memory 409600                    vcores 0,memory 0
    application_1482743319705_0006            root.SEQ.queueB   hbase    0                       1                           0                    vcores 0,memory 0                vcores 1,memory 1536                     vcores 0,memory 0

  • application -status <app_id> command displays detail information of specified application. Here is an example:

    > superior application -status application_1443067302606_0609
    ID: application_1443067302606_0609
    QUEUE: root.Q1.Q11
    USER: cchen
    RESOURCE_REQUEST: vcores 3,memory 3072
    RESOURCE_INUSE: vcores 2,memory 2048
    RESOURCE_RESERVED:vcores 1, memory 1024
    NUMBER_RUNNING_CONTAINER: 2
    NUMBER_PENDING_REQUEST: 3
    NUMBER_RESERVED_CONTAINER: 1
    MASTER_CONTAINER_ID: application_1443067302606_0609_01
    MASTER_CONTAINER_RESOURCE: node1.domain.com
    BLACKLIST: node5,node8
    DEMANDS:
    |-- PRIORITY: 20
    |-- MASTER: true
    |-- CAPABILITY: vcores 2, memory 2048
    |-- COUNT: 1
    |-- RESERVED_RES : vcores 1, memory 1024
    |-- RELAXLOCALITY: true
    |-- LOCALITY: node1/1
    |-- RESOURCESELECT: label1
    |-- PENDINGREASON: "application limit reached"
    |-- ID: application_1443067302606_0609_03
    |-- RESOURCE: node1.domain.com
    |-- RESERVED_RES: vcores 1, memory 1024
    |
    |--PRIORITY: 1
    |-- MASTER: false
    |-- CAPABILITY: vcores 1,memory 1024
    |-- COUNT: 2
    |-- RESERVED_RES: vcores 0, memory 0
    |-- RELAXLOCLITY: true
    |--LOCALITY: node1/1,node2/1,rackA/2
    |-- RESOURCESELECT: label1
    |-- PENDINGREASON: "no available resource"
    CONTAINERS:
    |-- ID: application_1443067302606_0609_01
    |-- RESOURCE: node1.domain.com
    |-- CAPABILITY: vcores 1,memory 1024
    |
    |-- ID: application_1443067302606_0609_02
    |-- RESOURCE: node2.domain.com
    |-- CAPABILITY: vcores 1,memory 1024

• Superior policy command

  This command displays policy related information.

  

  Superior policy command can be used only by admin user and user with yarn admin rigths.

  Usage output:

  >superior policy
  
  Usage: policy [-help]
                [-list <resourcepoolname>] [-u] [-detail]
                [-status <resourcepoolname>]
   -detail                      only work with -list option to show a
                                summary information of resource pool
                                distribution on queues, including reserve,
                                minimum and maximum
   -help                        prints policy sub command usage
   -list <resourcepoolname>     prints a summary information of resource
                                pool distribution on queue
   -status <resourcepoolname>   prints pool distribution policy
                                configuration and status of specified
                                resource pool
   -u                           only work with -list option to show a
                                summary information of resource pool
                                distribution on queues and also user
                                accounts

  • policy -list <resourcepoolname> print out a summary of queue distribution information. Here is an example:

    >superior policy -list default
    NAME: default
    TOTAL_RESOURCE: vcores 16,memory 16384
    AVAILABLE_RESOURCE: vcores 16,memory 16384
    
    NAMERES_INUSERES_REQUEST
    root.defaultvcores 0,memory 0vcores 0,memory 0
    root.productionvcores 0,memory 0vcores 0,memory 0
    root.production.BU1vcores 0,memory 0vcores 0,memory 0
    root.production.BU2 vcores 0,memory 0vcores 0,memory 0

  • policy -list <resourcepoolname> -u prints out also user level summary information

    > superior policy -list default -u
    NAME: default
    TOTAL_RESOURCE: vcores 16,memory 16384
    AVAILABLE_RESOURCE: vcores 16,memory 16384
    
    NAMERES_INUSERES_REQUEST
    root.defaultvcores 0,memory 0vcores 0,memory 0
    root.default.[_others_]vcores 0,memory 0vcores 0,memory 0
    root.productionvcores 0,memory 0vcores 0,memory 0
    root.production.BU1vcores 0,memory 0vcores 0,memory 0
    root.production.BU1.[_others_]vcores 0,memory 0vcores 0,memory 0
    root.production.BU2vcores 0,memory 0vcores 0,memory 0
    root.production.BU2.[_others_]vcores 0,memory 0vcores 0,memory 0

  • policy -status <resourcepoolname> print out policy detail of specified resource pool. Here is an example:

    > superior policy -status pool1
    NAME: pool1
    TOTAL_RESOURCE: vcores 64,memory 128
    AVAILABLE_RESOURCE: vcores 40,memory 28
    QUEUES:
    |-- NAME: root.Q1
    |-- RESOURCE_USE: vcores 20, memory 1000
    |-- RESOURCE_REQUEST: vcores 2,memory 100
    |--RESERVE: vcores 10, memory 4096
    |--MINIMUM: vcore 11, memory 4096
    |--MAXIMUM: vcores 500, memory 100000
    |--CONFIGURATION:
    |-- SHARE: 50%
    |-- RESERVE: vcores 10, memory 4096
    |-- MINIMUM: vcores 11, memory 4096
    |-- MAXIMUM: vcores 500, memory 100000
    |-- QUEUES:
    |-- NAME: root.Q1.Q11
    |-- RESOURCE_USE: vcores 15, memory, 500
    |-- RESOURCE_REQUEST: vcores 1, memory 50
    |-- RESERVE: vcores 0, memory 0
    |-- MINIMUM: vcores 0, memory 0
    |-- MAXIMUM: vcores -1, memory -1
    |-- USER_ACCOUNTS:
    |-- NAME: user1
    |-- RESOURCE_USE: vcores 1, memory 10
    |-- RESOURCE_REQUEST: vcores 1, memory 50
    |
    |-- NAME: OTHERS
    |--RESOURCE_USE: vcores 0, memory 0
    |- RESOURCE_REQUEST: vcores 0, memory 0
    |-- CONFIGURATION:
    |-- SHARE: 100%
    |-- USER_POLICY:
    |-- NAME: user1
    |-- WEIGHT: 10
    |
    |-- NAME: OTHERS
    |-- WEIGHT: 1
    |-- MAXIMUM: vcores 10, memory 1000