# YARN Command介绍 管理员可以使用**yarn**命令对YARN集群进行一些操作，例如启动ResourceManager、提交应用程序、中止应用、查询节点状态、下载container日志等操作。 #### YARN Command YARN Commands可同时供普通用户和管理员用户使用，它包含了少量普通用户可以执行的命令，比如jar、logs，而大部分只有管理员有权限使用。 用法： 进入Yarn客户端目录，执行命令导入环境变量，然后即可直接运行相关命令。 ``` source bigdata_env ``` ``` yarn [--config confdir] COMMAND ``` 常见命令功能介绍可参考[表1]，也可通过以下命令查看帮助信息。 ``` yarn --help ``` 更多完整和详细的命令描述可以参考官网文档：。 表1常用Command功能描述 | **COMMAND** | **描述** | |:---|:---| | resourcemanager | 启动ResourceManager进程。 以**omm** 用户执行服务端命令前需手动配置环境变量（客户端需要有**omm** 用户的执行权限），如下命令所示，**具体目录地址以集群ResourceManager节点为准**。 ``` export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_ResourceManager/etc export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_ResourceManager/etc ``` | | nodemanager | 运行NodeManager进程。 以**omm** 用户执行服务端命令前需手动配置环境变量（客户端需要有**omm** 用户的执行权限），如下命令所示，**具体目录地址以集群NodeManager节点为准**。 ``` export YARN_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_NodeManager/etc export HADOOP_CONF_DIR=${BIGDATA_HOME}/FusionInsight_HD_XXX/X_XX_NodeManager/etc ``` | | rmadmin | 用于对ResourceManager进行运行时配置调整、状态管理、HA 故障转移等操作。 | | version | 查看版本信息。 | | jar \ | 运行打包的YARN应用Jar包。 yarn jar \ \[主类\] \[参数...\] | | logs | 用于获取应用程序的聚合日志。 - -applicationId \：必选，指定要查看日志的应用程序ID。 - -containerId \：查看特定Container的日志。 - -nodeAddress \：查看特定节点上的日志，格式为nodename:port。 - -log_files \：过滤特定的日志文件（如只看stderr或stdout）。 - -show_application_log_info：显示日志的元数据信息（大小、位置等）而不打印内容。 | | classpath | 获取Hadoop JAR包和其他库文件所需的CLASSPATH路径。 | | daemonlog | 动态获取/设置守护进程的日志级别。 | | CLASSNAME | 运行自定义Java类。 | | top | 运行集群利用率监控工具，实时显示集群的资源使用情况（应用数量、内存/CPU 使用率）。 | | -Dmapreduce.job.hdfs-servers | 如果对接了OBS，而服务端依然使用HDFS，那么需要显式在命令行使用该参数指定HDFS的地址。 格式为hdfs://*{NAMESERVICE}*，其中{NAMESERVICE}为HDFS中的NameService名称，如果当前的HDFS具有多个NameService，那么需要指定所有的NameService，并以","隔开。 ``` hdfs://nameservice1,hdfs://nameservice2 ``` | #### Superior Scheduler Command Superior Scheduler引擎提供了输出Superior Scheduler引擎具体信息的CLI，为了执行Superior命令，需要使用"\/bin/superior"脚本。 以下为superior命令格式： ``` /bin/superior Usage: superior [COMMAND | -help] Where COMMAND is one of: resourcepool prints resource pool status queue prints queue status application prints application status policy prints policy status ``` - **superior resourcepool** 该命令显示Resource Pool和相关策略的相关状态以及配置信息。 ![](https://support.huaweicloud.com/devg3-mrs/public_sys-resources/notice_3.0-zh-cn.png) **superior resourcepool**命令仅用于管理员用户及拥有yarn管理权限的用户。 ``` >superior resourcepool Usage: resourcepool [-help] [-list] [-status ] -help prints resource pool usage -list prints all resource pool summary report -status prints status and configuration of specified resource pool ``` - **resourcepool -list** 显示Resource Pool摘要信息。 ``` > superior resourcepool -list NAME NUMBER_MEMBER TOTAL_RESOURCE AVAILABLE_RESOURCE Pool1 4 vcores 30,memory 1000 vcores 21,memory 80 Pool2 100 vcores 100,memory 12800 vcores 30,memory 1000 default 2 vcores 64,memory 128 vcores 40,memory 28 ``` - **resourcepool -status** *\* 以列表格式显示资源库详细信息。 ``` > superior resourcepool -status default NAME: default DESCRIPTION: System generated resource pool TOTAL_RESOURCE: vcores 64,memory 128 AVAILABLE_RESOURCE: vcores 40,memory 28 NUMBER_MEMBER: 2 MEMBERS: node1,node2 CONFIGURATION: |-- RESOURCE_SELECT: |__RESOURCES: ``` - **superior queue** 该命令输出分层队列信息。 用法输出： ``` Usage: queue [-help] [-list] [-e] [[-name ] [-r|-c]] [-status ] -c only work with -name option. If this option is used, command will print information of specified queue and its direct children. -e only work with -list or -list -name option. If this option is used, command will print effective state of specified queue and all of its descendants. -help prints queue sub command usage -list prints queue summary report. This option can work with -name and -r options. -name print specified queue, this can work with -r option. By default, it will print queue's own information. When -r is defined, command will print all of its descendant queues. When -c is defined, it will print its direct children queues. -r only work with -name option. If this option is used, command will print information of specified queue and all of its descendants. -status prints status of specified queue ``` - **queue -list** 以表格格式输出队列摘要信息，命令将基于队列分层样式输出信息。可通过SUBMIT ACL或ADMIN ACL的队列权限查看队列。 ``` > superior queue -list NAME STATE NRUN_APP NPEND_APP NRUN_CONTAINER NPEND_REQUEST RES_INUSE RES_REQUEST root OPEN|ACTIVE 10 20 100 200 vcores 100,memory 1000 vcores 200,memory 2000 root.Q1 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q1.Q11 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q1.Q12 CLOSE|INACTIVE 0 0 0 0 vcores 0,memory 0 vcores 0,memory 0 root.Q2 OPEN|INACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q2.Q21 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 ``` - **queue -list -name** *root.Q1* 只输出root.Q1队列信息。 ``` > superior queue -list -name root.Q1 NAME STATE NRUN_APP NPEND_APP NRUN_CONTAINER NPEND_REQUEST RES_INUSE RES_REQUEST root.Q1 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 ``` - **queue -list -name** *root.Q1* **-r** 输出root.Q1及其所有的分支队列信息。 ``` > superior queue -list -name root.Q1 -r NAME STATE NRUN_APP NPEND_APP NRUN_CONTAINER NPEND_REQUEST RES_INUSE RES_REQUEST root.Q1 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q1.Q11 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q1.Q12 CLOSE|INACTIVE 0 0 0 0 vcores 0,memory 0 vcores 0,memory 0 ``` - **queue -list -name** root**-c** 输出root及其直系子目录。 ``` > superior queue -list -name root -c NAME STATE NRUN_APP NPEND_APP NRUN_CONTAINER NPEND_REQUEST RES_INUSE RES_REQUEST root OPEN|ACTIVE 10 20 100 200 vcores 100,memory 1000 vcores 200,memory 2000 root.Q1 OPEN|ACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 root.Q2 OPEN|INACTIVE 5 10 50 100 vcores 50,memory 500 vcores 100,memory 1000 ``` - **queue -status** *\* 输出具体队列状态和配置。 用户可通过SUBMIT ACL权限查看除队列ACL外的细节信息，也可通过ADMIN ACL的队列权限查看包括ACL在内的队列细节信息。 ``` > superior queue -status root.Q1 NAME: root.Q1 OPEN_STATE:CLOSED ACTIVE_STATE: INACTIVE EOPEN_STATE: CLOSED EACTIVE_STATE: INACTIVE LEAF_QUEUE: Yes NUMBER_PENDING_APPLICATION: 100 NUMBER_RUNNING_APPLICATION: 10 NUMBER_PENDING_REQUEST: 10 NUMBER_RUNNING_CONTAINER: 10 NUMBER_RESERVED_CONTAINER: 0 RESOURCE_REQUEST: vcores 3,memory 3072 RESOURCE_INUSE: vcores 2,memory 2048 RESOURCE_RESERVED: vcores 0,memory 0 CONFIGURATION: |-- DESCRIPTION: Spark session queue |-- MAX_PENDING_APPLICATION: 10000 |--MAX_RUNNING_APPLICATION: 1000 |--ALLOCATION_ORDER_POLICY: FIFO |--DEFAULT_RESOURCE_SELECT: label1 |--MAX_MASTER_SHARE: 10% |--MAX_RUNNING_APPLICATION_PER_USER : -1 |--MAX_ALLOCATION_UNIT: vcores 32,memory 12800 |--ACL_USERS: user1,user2 |--ACL_USERGROUPS: usergroup1,usergroup2 |-- ACL_ADMINS: user1 |--ACL_ADMINGROUPS: usergroup1 ``` - **superior application** 该命令输出应用相关信息。 ``` >superior application Usage: application [-help] [-list] [-status ] -help prints application sub command usage -list prints all application summary report -status prints status of specified application ``` 用户可通过应用的浏览访问权限查看应用相关信息。 - **application -list** 以表的形式提供所有应用的信息摘要。 ``` > superior application -list ID QUEUE USER NRUN_CONTAINER NPEND_REQUEST NRSV_CONTAINER RES_INUSE RES_REQUEST RES_RESERVED application_1482743319705_0005 root.SEQ.queueB hbase 1 100 0 vcores 1,memory 1536 vcores 2000,memory 409600 vcores 0,memory 0 application_1482743319705_0006 root.SEQ.queueB hbase 0 1 0 vcores 0,memory 0 vcores 1,memory 1536 vcores 0,memory 0 ``` - **application -status** *\* 输出指定应用的详细信息。 ``` > superior application -status application_1443067302606_0609 ID: application_1443067302606_0609 QUEUE: root.Q1.Q11 USER: cchen RESOURCE_REQUEST: vcores 3,memory 3072 RESOURCE_INUSE: vcores 2,memory 2048 RESOURCE_RESERVED:vcores 1, memory 1024 NUMBER_RUNNING_CONTAINER: 2 NUMBER_PENDING_REQUEST: 3 NUMBER_RESERVED_CONTAINER: 1 MASTER_CONTAINER_ID: application_1443067302606_0609_01 MASTER_CONTAINER_RESOURCE: node1.domain.com BLACKLIST: node5,node8 DEMANDS: |-- PRIORITY: 20 |-- MASTER: true |-- CAPABILITY: vcores 2, memory 2048 |-- COUNT: 1 |-- RESERVED_RES : vcores 1, memory 1024 |-- RELAXLOCALITY: true |-- LOCALITY: node1/1 |-- RESOURCESELECT: label1 |-- PENDINGREASON: “application limit reached” |-- ID: application_1443067302606_0609_03 |-- RESOURCE: node1.domain.com |-- RESERVED_RES: vcores 1, memory 1024 | |--PRIORITY: 1 |-- MASTER: false |-- CAPABILITY: vcores 1,memory 1024 |-- COUNT: 2 |-- RESERVED_RES: vcores 0, memory 0 |-- RELAXLOCLITY: true |--LOCALITY: node1/1,node2/1,rackA/2 |-- RESOURCESELECT: label1 |-- PENDINGREASON: “no available resource” CONTAINERS: |-- ID: application_1443067302606_0609_01 |-- RESOURCE: node1.domain.com |-- CAPABILITY: vcores 1,memory 1024 | |-- ID: application_1443067302606_0609_02 |-- RESOURCE: node2.domain.com |-- CAPABILITY: vcores 1,memory 1024 ``` - **superior policy** 该命令输出决策相关信息。 ![](https://support.huaweicloud.com/devg3-mrs/public_sys-resources/notice_3.0-zh-cn.png) **superior policy**命令仅用于管理员用户及拥有yarn管理权限的用户。 ``` >superior policy Usage: policy [-help] [-list ] [-u] [-detail] [-status ] -detail only work with -list option to show a summary information of resource pool distribution on queues, including reserve, minimum and maximum -help prints policy sub command usage -list prints a summary information of resource pool distribution on queue -status prints pool distribution policy configuration and status of specified resource pool -u only work with -list option to show a summary information of resource pool distribution on queues and also user accounts ``` - **policy -list** *\* 输出队列分布信息摘要。 ``` >superior policy -list default NAME: default TOTAL_RESOURCE: vcores 16,memory 16384 AVAILABLE_RESOURCE: vcores 16,memory 16384 NAMERES_INUSERES_REQUEST root.defaultvcores 0,memory 0vcores 0,memory 0 root.productionvcores 0,memory 0vcores 0,memory 0 root.production.BU1vcores 0,memory 0vcores 0,memory 0 root.production.BU2 vcores 0,memory 0vcores 0,memory 0 ``` - **policy -list** *\* **-u** 输出用户级信息摘要。 ``` > superior policy -list default -u NAME: default TOTAL_RESOURCE: vcores 16,memory 16384 AVAILABLE_RESOURCE: vcores 16,memory 16384 NAMERES_INUSERES_REQUEST root.defaultvcores 0,memory 0vcores 0,memory 0 root.default.[_others_]vcores 0,memory 0vcores 0,memory 0 root.productionvcores 0,memory 0vcores 0,memory 0 root.production.BU1vcores 0,memory 0vcores 0,memory 0 root.production.BU1.[_others_]vcores 0,memory 0vcores 0,memory 0 root.production.BU2vcores 0,memory 0vcores 0,memory 0 root.production.BU2.[_others_]vcores 0,memory 0vcores 0,memory 0 ``` - **policy -status** *\* 输出指定资源库的策略详细信息。 ``` > superior policy -status pool1 NAME: pool1 TOTAL_RESOURCE: vcores 64,memory 128 AVAILABLE_RESOURCE: vcores 40,memory 28 QUEUES: |-- NAME: root.Q1 |-- RESOURCE_USE: vcores 20, memory 1000 |-- RESOURCE_REQUEST: vcores 2,memory 100 |--RESERVE: vcores 10, memory 4096 |--MINIMUM: vcore 11, memory 4096 |--MAXIMUM: vcores 500, memory 100000 |--CONFIGURATION: |-- SHARE: 50% |-- RESERVE: vcores 10, memory 4096 |-- MINIMUM: vcores 11, memory 4096 |-- MAXIMUM: vcores 500, memory 100000 |-- QUEUES: |-- NAME: root.Q1.Q11 |-- RESOURCE_USE: vcores 15, memory, 500 |-- RESOURCE_REQUEST: vcores 1, memory 50 |-- RESERVE: vcores 0, memory 0 |-- MINIMUM: vcores 0, memory 0 |-- MAXIMUM: vcores -1, memory -1 |-- USER_ACCOUNTS: |-- NAME: user1 |-- RESOURCE_USE: vcores 1, memory 10 |-- RESOURCE_REQUEST: vcores 1, memory 50 | |-- NAME: OTHERS |--RESOURCE_USE: vcores 0, memory 0 |- RESOURCE_REQUEST: vcores 0, memory 0 |-- CONFIGURATION: |-- SHARE: 100% |-- USER_POLICY: |-- NAME: user1 |-- WEIGHT: 10 | |-- NAME: OTHERS |-- WEIGHT: 1 |-- MAXIMUM: vcores 10, memory 1000 ```