Spark REST API接口介绍
功能简介
Spark的REST API以JSON格式展现Web UI的一些指标,提供用户一种更简单的方法去创建新的展示和监控的工具,并且支持查询正在运行的app和已经结束的app的相关信息。开源的Spark REST接口支持对Jobs、Stages、Storage、Environment和Executors的信息进行查询,FusionInsight版本中添加了查询SQL、JDBC Server和Streaming的信息的REST接口。开源REST接口完整和详细的描述请参考官网上的文档以了解其使用方法:https://spark.apache.org/docs/3.1.1/monitoring.html#rest-api。
准备运行环境
安装客户端。在节点上安装客户端,如安装到“/opt/client”目录。
REST接口
通过以下命令可跳过REST接口过滤器获取相应的应用信息。
![](https://support.huaweicloud.com/devg3-mrs/public_sys-resources/notice_3.0-zh-cn.png)
- 安全模式下,JobHistory仅支持https协议,故在如下命令的url中请使用https协议。
- 安全模式下,需要设置spark.ui.customErrorPage=false并重启spark2x服务 (JobHistory2x、JDBCServer2x和SparkResource2x三个实例对应的参数都需要修改)。
![](https://support.huaweicloud.com/devg3-mrs/public_sys-resources/note_3.0-zh-cn.png)
- 下载curl安装包(http://curl.haxx.se/download/)。
- 使用如下命令进行安装包解压:
- 使用如下命令覆盖安装:
./configure
make
make install
- 使用如下命令更新curl的动态链接库:
- 安装成功后,重新登录节点环境,使用如下命令查看curl版本是否更新成功:
- 获取JobHistory中所有应用信息:
- 命令:
curl -k -i --negotiate -u: "https://192.168.227.16:18080/api/v1/applications"
其中192.168.227.16为JobHistory节点的业务IP,18080为JobHistory的端口号。
- 结果:
[ { "id" : "application_1517290848707_0008", "name" : "Spark Pi", "attempts" : [ { "startTime" : "2018-01-30T15:05:37.433CST", "endTime" : "2018-01-30T15:06:04.625CST", "lastUpdated" : "2018-01-30T15:06:04.848CST", "duration" : 27192, "sparkUser" : "sparkuser", "completed" : true, "startTimeEpoch" : 1517295937433, "endTimeEpoch" : 1517295964625, "lastUpdatedEpoch" : 1517295964848 } ] }, { " id" : "application_1517290848707_0145", "name" : "Spark shell", "attempts" : [ { "startTime" : "2018-01-31T15:20:31.286CST", "endTime" : "1970-01-01T07:59:59.999CST", "lastUpdated" : "2018-01-31T15:20:47.086CST", "duration" : 0, "sparkUser" : "admintest", "completed" : false, "startTimeEpoch" : 1517383231286, "endTimeEpoch" : -1, "lastUpdatedEpoch" : 1517383247086 } ] }]
- 结果分析:
- 命令:
- 获取JobHistory中某个应用的信息:
- 命令:
curl -k -i --negotiate -u: "https://192.168.227.16:18080/api/v1/applications/application_1517290848707_0008"
其中192.168.227.16为JobHistory节点的业务IP,18080为JobHistory的端口号,application_1517290848707_0008为应用的id。
- 结果:
{ "id" : "application_1517290848707_0008", "name" : "Spark Pi", "attempts" : [ { "startTime" : "2018-01-30T15:05:37.433CST", "endTime" : "2018-01-30T15:06:04.625CST", "lastUpdated" : "2018-01-30T15:06:04.848CST", "duration" : 27192, "sparkUser" : "sparkuser", "completed" : true, "startTimeEpoch" : 1517295937433, "endTimeEpoch" : 1517295964625, "lastUpdatedEpoch" : 1517295964848 } ] }
- 结果分析:
通过这个命令,可以查询某个Spark应用的信息,显示的信息如表1所示。
- 命令:
- 获取正在执行的某个应用的Executor信息:
- 针对alive executor命令:
curl -k -i --negotiate -u: "https://192.168.169.84:8090/proxy/application_1478570725074_0046/api/v1/applications/application_1478570725074_0046/executors"
- 针对全部executor(alive&dead)命令:
curl -k -i --negotiate -u: "https://192.168.169.84:8090/proxy/application_1478570725074_0046/api/v1/applications/application_1478570725074_0046/allexecutors"
其中192.168.169.84为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1478570725074_0046为在YARN中的应用ID。
- 结果:
[{ "id" : "driver", "hostPort" : "192.168.169.84:23886", "isActive" : true, "rddBlocks" : 0, "memoryUsed" : 0, "diskUsed" : 0, "activeTasks" : 0, "failedTasks" : 0, "completedTasks" : 0, "totalTasks" : 0, "totalDuration" : 0, "totalInputBytes" : 0, "totalShuffleRead" : 0, "totalShuffleWrite" : 0, "maxMemory" : 278019440, "executorLogs" : { } }, { "id" : "1", "hostPort" : "192.168.169.84:23902", "isActive" : true, "rddBlocks" : 0, "memoryUsed" : 0, "diskUsed" : 0, "totalCores" : 1, "maxTasks" : 1, "activeTasks" : 0, "failedTasks" : 0, "completedTasks" : 0, "totalTasks" : 0, "totalDuration" : 0, "totalGCTime" : 139, "totalInputBytes" : 0, "totalShuffleRead" : 0, "totalShuffleWrite" : 0, "maxMemory" : 555755765, "executorLogs" : { "stdout" : "https://XTJ-224:8044/node/containerlogs/container_1478570725074_0049_01_000002/admin/stdout?start=-4096", "stderr" : "https://XTJ-224:8044/node/containerlogs/container_1478570725074_0049_01_000002/admin/stderr?start=-4096" } } ]
- 结果分析:
- 针对alive executor命令:
REST API增强
- SQL相关的命令:获取所有SQL语句和执行时间最长的SQL语句
- SparkUI命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1476947670799_0053为在YARN中的应用ID。
可以在命令后的url路径增加相应的参数设置,搜索对应的SQL语句。
例如,查看100条sql语句:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL?limit=100"
查看正在运行的参数:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL?completed=false"
- JobHistory命令:
curl -k -i --negotiate -u: "https://192.168.227.16:18080/api/v1/applications/application_1478570725074_0004/SQL"
其中192.168.227.16为JobHistory节点的业务IP,18080为JobHistory的端口号,application_1478570725074_0004为应用ID。
- 结果:
SparkUI命令和JobHistory命令的查询结果均为:
{ "longestDurationOfCompletedSQL" : [ { "id" : 0, "status" : "COMPLETED", "description" : "getCallSite at SQLExecution.scala:48", "submissionTime" : "2016/11/08 15:39:00", "duration" : "2 s", "runningJobs" : [ ], "successedJobs" : [ 0 ], "failedJobs" : [ ] } ], "sqls" : [ { "id" : 0, "status" : "COMPLETED", "description" : "getCallSite at SQLExecution.scala:48", "submissionTime" : "2016/11/08 15:39:00", "duration" : "2 s", "runningJobs" : [ ], "successedJobs" : [ 0 ], "failedJobs" : [ ] }] }
- 结果分析:
通过这个命令,可以查询当前应用的所有SQL语句的信息(即结果中“sqls”的部分),执行时间最长的SQL语句的信息(即结果中“longestDurationOfCompletedSQL”的部分)。每个SQL语句的信息如下表3。
- SparkUI命令:
- JDBC Server相关的命令:获取连接数,正在执行的SQL数,所有session信息,所有SQL的信息
- 命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/sqlserver"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1476947670799_0053为在YARN中的应用ID。
- 结果:
{ "sessionNum" : 1, "runningSqlNum" : 0, "sessions" : [ { "user" : "spark", "ip" : "192.168.169.84", "sessionId" : "9dfec575-48b4-4187-876a-71711d3d7a97", "startTime" : "2016/10/29 15:21:10", "finishTime" : "", "duration" : "1 minute 50 seconds", "totalExecute" : 1 } ], "sqls" : [ { "user" : "spark", "jobId" : [ ], "groupId" : "e49ff81a-230f-4892-a209-a48abea2d969", "startTime" : "2016/10/29 15:21:13", "finishTime" : "2016/10/29 15:21:14", "duration" : "555 ms", "statement" : "show tables", "state" : "FINISHED", "detail" : "== Parsed Logical Plan ==\nShowTablesCommand None\n\n== Analyzed Logical Plan ==\ntableName: string, isTemporary: boolean\nShowTablesCommand None\n\n== Cached Logical Plan ==\nShowTablesCommand None\n\n== Optimized Logical Plan ==\nShowTablesCommand None\n\n== Physical Plan ==\nExecutedCommand ShowTablesCommand None\n\nCode Generation: true" } ] }
- 结果分析:
- 命令:
- JDBC api增强通过beeline里面获取的executionID 取消当前正在执行的SQL
- Streaming相关的命令:获取平均输入频率,平均调度时延,平均执行时长,总时延平均值
- 命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1477722033672_0008/api/v1/applications/application_1477722033672_0008/streaming/statistics"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1477722033672_0008为在YARN中的应用ID。
- 结果:
{ "startTime" : "2018-12-25T08:58:10.836GMT", "batchDuration" : 1000, "numReceivers" : 1, "numActiveReceivers" : 1, "numInactiveReceivers" : 0, "numTotalCompletedBatches" : 373, "numRetainedCompletedBatches" : 373, "numActiveBatches" : 0, "numProcessedRecords" : 1, "numReceivedRecords" : 1, "avgInputRate" : 0.002680965147453083, "avgSchedulingDelay" : 14, "avgProcessingTime" : 47, "avgTotalDelay" : 62 }
- 结果分析:
通过这个命令,可以查询当前Streaming应用的平均输入频率(events/sec),平均调度时延(ms),平均执行时长(ms),总时延平均值(ms)。
- 命令: