更新时间:2025-12-24 GMT+08:00
分享

YARN Command介绍

管理员可以使用yarn命令对YARN集群进行一些操作,例如启动ResourceManager、提交应用程序、中止应用、查询节点状态、下载container日志等操作。

YARN Command

YARN Commands可同时供普通用户和管理员用户使用,它包含了少量普通用户可以执行的命令,比如jar、logs,而大部分只有管理员有权限使用。

用法:

进入Yarn客户端目录,执行命令导入环境变量,然后即可直接运行相关命令。

source bigdata_env
yarn [--config confdir] COMMAND

常见命令功能介绍可参考表1,也可通过以下命令查看帮助信息。

yarn --help

更多完整和详细的命令描述可以参考官网文档:http://hadoop.apache.org/docs/r3.3.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html

表1 常用Command功能描述

COMMAND

描述

resourcemanager

启动ResourceManager进程。

omm用户执行服务端命令前需手动配置环境变量(客户端需要有omm用户的执行权限),如下命令所示,具体目录地址以集群ResourceManager节点为准

export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_ResourceManager/etc
export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_ResourceManager/etc

nodemanager

运行NodeManager进程。

omm用户执行服务端命令前需手动配置环境变量(客户端需要有omm用户的执行权限),如下命令所示,具体目录地址以集群NodeManager节点为准

export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_NodeManager/etc
export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_NodeManager/etc

rmadmin

用于对ResourceManager进行运行时配置调整、状态管理、HA 故障转移等操作。

version

查看版本信息。

jar <jar>

运行打包的YARN应用Jar包。

yarn jar <jar文件> [主类] [参数...]

logs

用于获取应用程序的聚合日志。

  • -applicationId <ApplicationId>:必选,指定要查看日志的应用程序ID。
  • -containerId <ContainerId>:查看特定Container的日志。
  • -nodeAddress <NodeAddress>:查看特定节点上的日志,格式为nodename:port。
  • -log_files <LogFilePattern>:过滤特定的日志文件(如只看stderr或stdout)。
  • -show_application_log_info:显示日志的元数据信息(大小、位置等)而不打印内容。

classpath

获取Hadoop JAR包和其他库文件所需的CLASSPATH路径。

daemonlog

动态获取/设置守护进程的日志级别。

CLASSNAME

运行自定义Java类。

top

运行集群利用率监控工具,实时显示集群的资源使用情况(应用数量、内存/CPU 使用率)。

-Dmapreduce.job.hdfs-servers

如果对接了OBS,而服务端依然使用HDFS,那么需要显式在命令行使用该参数指定HDFS的地址。

格式为hdfs://{NAMESERVICE},其中{NAMESERVICE}为HDFS中的NameService名称,如果当前的HDFS具有多个NameService,那么需要指定所有的NameService,并以“,”隔开。

hdfs://nameservice1,hdfs://nameservice2

Superior Scheduler Command

Superior Scheduler引擎提供了输出Superior Scheduler引擎具体信息的CLI,为了执行Superior命令,需要使用“<HADOOP_HOME>/bin/superior”脚本。

以下为superior命令格式:

<HADOOP_HOME>/bin/superior
Usage: superior [COMMAND | -help]
   Where COMMAND is one of:
   resourcepool                        prints resource pool status
   queue                               prints queue status
   application                         prints application status
   policy                              prints policy status
  • superior resourcepool

    该命令显示Resource Pool和相关策略的相关状态以及配置信息。

    superior resourcepool命令仅用于管理员用户及拥有yarn管理权限的用户。

    >superior resourcepool
    Usage: resourcepool [-help]
                        [-list]
                        [-status <resourcepoolname>]
     -help                        prints resource pool usage
     -list                        prints all resource pool summary report
     -status <resourcepoolname>   prints status and configuration of specified
                                  resource pool
    • resourcepool -list
      显示Resource Pool摘要信息。
      > superior resourcepool -list
      NAME       NUMBER_MEMBER     TOTAL_RESOURCE           AVAILABLE_RESOURCE
      Pool1      4                 vcores 30,memory 1000    vcores 21,memory 80
      Pool2      100               vcores 100,memory 12800  vcores 30,memory 1000
      default    2                 vcores 64,memory 128     vcores 40,memory 28
    • resourcepool -status <resourcepoolname>
      以列表格式显示资源库详细信息。
      > superior resourcepool -status default
      NAME: default
      DESCRIPTION: System generated resource pool
      TOTAL_RESOURCE: vcores 64,memory 128
      AVAILABLE_RESOURCE: vcores 40,memory 28
      NUMBER_MEMBER: 2
      MEMBERS: node1,node2
      CONFIGURATION:
      |-- RESOURCE_SELECT:
      |__RESOURCES:
  • superior queue

    该命令输出分层队列信息。

    用法输出:
    Usage: queue [-help]
                 [-list] [-e] [[-name <queue_name>] [-r|-c]]
                 [-status <queue_name>]
     -c                     only work with -name <queue_name> option. If this
                            option is used, command  will print information of
                            specified queue and its direct children.
     -e                     only work with -list or -list -name option. If
                            this option is used, command will print effective
                            state of specified queue and all of its
                            descendants.
     -help                  prints queue sub command usage
     -list                  prints queue summary report. This option can work
                            with -name <queue_name> and -r options.
     -name <queue_name>     print specified queue, this can work with -r
                            option. By default, it will print queue's own
                            information. When -r is defined, command will
                            print all of its descendant queues. When -c is
                            defined, it will print its direct children queues.
     -r                     only work with -name <queue_name> option. If this
                            option is used, command will print information of
                            specified queue and all of its descendants.
     -status <queue_name>   prints status of specified queue
    • queue -list
      以表格格式输出队列摘要信息,命令将基于队列分层样式输出信息。可通过SUBMIT ACL或ADMIN ACL的队列权限查看队列。
      > superior queue -list
      NAME         STATE            NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                 RES_REQUEST
      root         OPEN|ACTIVE      10           20            100              200              vcores 100,memory 1000    vcores 200,memory 2000
      root.Q1      OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q1.Q11  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q1.Q12  CLOSE|INACTIVE   0            0             0                0                vcores 0,memory 0         vcores 0,memory 0
      root.Q2      OPEN|INACTIVE    5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
      root.Q2.Q21  OPEN|ACTIVE      5            10            50               100              vcores 50,memory 500      vcores 100,memory 1000
    • queue -list -name root.Q1
      只输出root.Q1队列信息。
      > superior queue -list -name root.Q1
      NAME            STATE           NRUN_APP     NPEND_APP      NRUN_CONTAINER  NPEND_REQUEST    RES_INUSE            RES_REQUEST
      root.Q1         OPEN|ACTIVE     5            10             50              100              vcores 50,memory 500 vcores 100,memory 1000
    • queue -list -name root.Q1 -r
      输出root.Q1及其所有的分支队列信息。
      > superior queue -list -name root.Q1 -r
      NAME         STATE            NRUN_APP     NPEND_APP  NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE               RES_REQUEST
      root.Q1      OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
      root.Q1.Q11  OPEN|ACTIVE      5            10         50               100              vcores 50,memory 500    vcores 100,memory 1000
      root.Q1.Q12  CLOSE|INACTIVE   0            0          0                0                vcores 0,memory 0       vcores 0,memory 0
    • queue -list -name root -c
      输出root及其直系子目录。
      > superior queue -list -name root -c
      NAME                  STATE             NRUN_APP     NPEND_APP     NRUN_CONTAINER   NPEND_REQUEST    RES_INUSE                RES_REQUEST
      root                  OPEN|ACTIVE       10           20            100              200              vcores 100,memory 1000   vcores 200,memory 2000
      root.Q1               OPEN|ACTIVE       5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000
      root.Q2               OPEN|INACTIVE     5            10            50               100              vcores 50,memory 500     vcores 100,memory 1000
    • queue -status <queue_name>

      输出具体队列状态和配置。

      用户可通过SUBMIT ACL权限查看除队列ACL外的细节信息,也可通过ADMIN ACL的队列权限查看包括ACL在内的队列细节信息。

      > superior queue -status root.Q1
      NAME: root.Q1
      OPEN_STATE:CLOSED
      ACTIVE_STATE: INACTIVE
      EOPEN_STATE: CLOSED
      EACTIVE_STATE: INACTIVE
      LEAF_QUEUE: Yes
      NUMBER_PENDING_APPLICATION: 100
      NUMBER_RUNNING_APPLICATION: 10
      NUMBER_PENDING_REQUEST: 10
      NUMBER_RUNNING_CONTAINER: 10
      NUMBER_RESERVED_CONTAINER: 0
      RESOURCE_REQUEST: vcores 3,memory 3072
      RESOURCE_INUSE: vcores 2,memory 2048
      RESOURCE_RESERVED: vcores 0,memory 0
      CONFIGURATION:
      |-- DESCRIPTION: Spark session queue
      |-- MAX_PENDING_APPLICATION: 10000
      |--MAX_RUNNING_APPLICATION: 1000
      |--ALLOCATION_ORDER_POLICY: FIFO
      |--DEFAULT_RESOURCE_SELECT: label1
      |--MAX_MASTER_SHARE: 10%
      |--MAX_RUNNING_APPLICATION_PER_USER : -1
      |--MAX_ALLOCATION_UNIT: vcores 32,memory 12800
      |--ACL_USERS: user1,user2
      |--ACL_USERGROUPS: usergroup1,usergroup2
      |-- ACL_ADMINS: user1
      |--ACL_ADMINGROUPS: usergroup1
  • superior application

    该命令输出应用相关信息。

    >superior application 
    Usage: application [-help]
                       [-list]
                       [-status <application_id>]
     -help                      prints application sub command usage
     -list                      prints all application summary report
     -status <application_id>   prints status of specified application

    用户可通过应用的浏览访问权限查看应用相关信息。

    • application -list
      以表的形式提供所有应用的信息摘要。
      > superior application -list
      ID                                        QUEUE             USER     NRUN_CONTAINER          NPEND_REQUEST               NRSV_CONTAINER       RES_INUSE                        RES_REQUEST                     RES_RESERVED           
      application_1482743319705_0005            root.SEQ.queueB   hbase    1                       100                         0                    vcores 1,memory 1536             vcores 2000,memory 409600       vcores 0,memory 0
      application_1482743319705_0006            root.SEQ.queueB   hbase    0                       1                           0                    vcores 0,memory 0                vcores 1,memory 1536            vcores 0,memory 0
    • application -status <application_id>
      输出指定应用的详细信息。
      > superior application -status application_1443067302606_0609
      ID: application_1443067302606_0609
      QUEUE: root.Q1.Q11
      USER: cchen
      RESOURCE_REQUEST: vcores 3,memory 3072
      RESOURCE_INUSE: vcores 2,memory 2048
      RESOURCE_RESERVED:vcores 1, memory 1024
      NUMBER_RUNNING_CONTAINER: 2
      NUMBER_PENDING_REQUEST: 3
      NUMBER_RESERVED_CONTAINER: 1
      MASTER_CONTAINER_ID: application_1443067302606_0609_01
      MASTER_CONTAINER_RESOURCE: node1.domain.com
      BLACKLIST: node5,node8
      DEMANDS:
      |-- PRIORITY: 20
      |-- MASTER: true
      |-- CAPABILITY: vcores 2, memory 2048
      |-- COUNT: 1
      |-- RESERVED_RES : vcores 1, memory 1024
      |-- RELAXLOCALITY: true
      |-- LOCALITY: node1/1
      |-- RESOURCESELECT: label1
      |-- PENDINGREASON: “application limit reached”
      |-- ID: application_1443067302606_0609_03
      |-- RESOURCE: node1.domain.com
      |-- RESERVED_RES: vcores 1, memory 1024
      |
      |--PRIORITY: 1
      |-- MASTER: false
      |-- CAPABILITY: vcores 1,memory 1024
      |-- COUNT: 2
      |-- RESERVED_RES: vcores 0, memory 0
      |-- RELAXLOCLITY: true
      |--LOCALITY: node1/1,node2/1,rackA/2
      |-- RESOURCESELECT: label1
      |-- PENDINGREASON: “no available resource”
      CONTAINERS:
      |-- ID: application_1443067302606_0609_01
      |-- RESOURCE: node1.domain.com
      |-- CAPABILITY: vcores 1,memory 1024
      |
      |-- ID: application_1443067302606_0609_02
      |-- RESOURCE: node2.domain.com
      |-- CAPABILITY: vcores 1,memory 1024
  • superior policy

    该命令输出决策相关信息。

    superior policy命令仅用于管理员用户及拥有yarn管理权限的用户。

    >superior policy
    Usage: policy [-help]
                  [-list <resourcepoolname>] [-u] [-detail]
                  [-status <resourcepoolname>]
     -detail                      only work with -list option to show a
                                  summary information of resource pool
                                  distribution on queues, including reserve,
                                  minimum and maximum
     -help                        prints policy sub command usage
     -list <resourcepoolname>     prints a summary information of resource
                                  pool distribution on queue
     -status <resourcepoolname>   prints pool distribution policy
                                  configuration and status of specified
                                  resource pool
     -u                           only work with -list option to show a
                                  summary information of resource pool
                                  distribution on queues and also user
                                  accounts
    • policy -list <resourcepoolname>
      输出队列分布信息摘要。
      >superior policy -list default
      NAME: default
      TOTAL_RESOURCE: vcores 16,memory 16384
      AVAILABLE_RESOURCE: vcores 16,memory 16384
      
      NAMERES_INUSERES_REQUEST
      root.defaultvcores 0,memory 0vcores 0,memory 0
      root.productionvcores 0,memory 0vcores 0,memory 0
      root.production.BU1vcores 0,memory 0vcores 0,memory 0
      root.production.BU2 vcores 0,memory 0vcores 0,memory 0
    • policy -list <resourcepoolname> -u
      输出用户级信息摘要。
      > superior policy -list default -u
      NAME: default
      TOTAL_RESOURCE: vcores 16,memory 16384
      AVAILABLE_RESOURCE: vcores 16,memory 16384
      
      NAMERES_INUSERES_REQUEST
      root.defaultvcores 0,memory 0vcores 0,memory 0
      root.default.[_others_]vcores 0,memory 0vcores 0,memory 0
      root.productionvcores 0,memory 0vcores 0,memory 0
      root.production.BU1vcores 0,memory 0vcores 0,memory 0
      root.production.BU1.[_others_]vcores 0,memory 0vcores 0,memory 0
      root.production.BU2vcores 0,memory 0vcores 0,memory 0
      root.production.BU2.[_others_]vcores 0,memory 0vcores 0,memory 0
    • policy -status <resourcepoolname>
      输出指定资源库的策略详细信息。
      > superior policy -status pool1
      NAME: pool1
      TOTAL_RESOURCE: vcores 64,memory 128
      AVAILABLE_RESOURCE: vcores 40,memory 28
      QUEUES:
      |-- NAME: root.Q1
      |-- RESOURCE_USE: vcores 20, memory 1000
      |-- RESOURCE_REQUEST: vcores 2,memory 100
      |--RESERVE: vcores 10, memory 4096
      |--MINIMUM: vcore 11, memory 4096
      |--MAXIMUM: vcores 500, memory 100000
      |--CONFIGURATION:
      |-- SHARE: 50%
      |-- RESERVE: vcores 10, memory 4096
      |-- MINIMUM: vcores 11, memory 4096
      |-- MAXIMUM: vcores 500, memory 100000
      |-- QUEUES:
      |-- NAME: root.Q1.Q11
      |-- RESOURCE_USE: vcores 15, memory, 500
      |-- RESOURCE_REQUEST: vcores 1, memory 50
      |-- RESERVE: vcores 0, memory 0
      |-- MINIMUM: vcores 0, memory 0
      |-- MAXIMUM: vcores -1, memory -1
      |-- USER_ACCOUNTS:
      |-- NAME: user1
      |-- RESOURCE_USE: vcores 1, memory 10
      |-- RESOURCE_REQUEST: vcores 1, memory 50
      |
      |-- NAME: OTHERS
      |--RESOURCE_USE: vcores 0, memory 0
      |- RESOURCE_REQUEST: vcores 0, memory 0
      |-- CONFIGURATION:
      |-- SHARE: 100%
      |-- USER_POLICY:
      |-- NAME: user1
      |-- WEIGHT: 10
      |
      |-- NAME: OTHERS
      |-- WEIGHT: 1
      |-- MAXIMUM: vcores 10, memory 1000

相关文档