Updated on 2024-08-16 GMT+08:00

Spark REST APIs

Function Description

Spark REST APIs display some metrics of the web UI in JSON format, provide users with a simpler method to create new display and monitoring tools, and enable users to query information about running apps and the completed apps. The open source Spark REST APIs allow users to query information about Jobs, Stages, Storage, Environment, and Executors. The REST APIs for querying the information about SQL, JDBC/ODBC Server, and Streaming are added to MRS. For details about open source RESTful APIs, visit https://spark.apache.org/docs/2.2.2/monitoring.html#rest-api.

Preparing an Operating Environment

Install a client in a directory on a node, for example, /opt/client.

  1. You have installed Spark on the server and confirmed that Spark is running properly.
  2. You have installed JDK 1.7 or 1.8 on the client operating environment.
  3. Obtain the MRS_Spark_Client.tar installation package, and run the following commands to decompress the package:

    tar -xvf MRS_Spark_Client.tar

    tar -xvf MRS_Spark_ClientConfig.tar

    You are advised to install a client of the same version as the cluster on the server to avoid version incompatibility.

  4. Go to the MRS_Spark_ClientConfig decompressed folder and run the following command to install the client:

    sh install.sh /opt/client

    In the preceding command, /opt/client is an example user-defined path.

  5. Go to /opt/client (the client installation directory) and run the following command to initialize environment variables:

    source bigdata_env

REST APIs

You can run the following command to skip the REST API filter to obtain application information:

  • Obtaining information about all applications in JobHistory
    • Command:
      curl https://192.168.227.16:18080/api/v1/applications?mode=monitoring --insecure

      192.168.227.16 is the service IP address of the JobHistory node, and 18080 is the port number of the JobHistory node.

    • Command output:
      [ {
      "id" : "application_1478570725074_0042",
      "name" : "Spark-JDBCServer",
      "attempts" : [ {
      "startTime" : "2016-11-09T16:57:15.237CST",
      "endTime" : "2016-11-09T17:01:22.573CST",
      "lastUpdated" : "2016-11-09T17:01:22.614CST",
      "duration" : 247336,
      "sparkUser" : "spark",
      "completed" : true
      } ]
      }, {
      "id" : "application_1478570725074_0047-part1",
      "name" : "SparkSQL::192.168.169.84",
      "attempts" : [ {
      "startTime" : "2016-11-10T11:57:36.626CST",
      "endTime" : "1969-12-31T07:59:59.999CST",
      "lastUpdated" : "2016-11-10T11:57:48.613CST",
      "duration" : 0,
      "sparkUser" : "admin",
      "completed" : false
      } ]
      }]
    • Result analysis:

      By running this command, you can query all Spark applications (including running applications and completed applications) in the current cluster. Table 1 provides information about the applications.

      Table 1 Basic application information

      Parameter

      Description

      id

      Application ID

      name

      Application name

      attempts

      Attempts executed by the application, including the attempt start time, attempt end time, user who initiates the attempts, and status indicating whether the attempts are completed.

  • Obtaining information about an application in JobHistory
    • Command:
      curl https://192.168.227.16:18080/api/v1/applications/application_1478570725074_0042?mode=monitoring --insecure

      192.168.227.16 is the service IP address of the JobHistory node, and 18080 is the port number of the JobHistory node. application_1478570725074_0042 is the application ID.

    • Command output:
      {
      "id" : "application_1478570725074_0042",
      "name" : "Spark-JDBCServer",
      "attempts" : [ {
      "startTime" : "2016-11-09T16:57:15.237CST",
      "endTime" : "2016-11-09T17:01:22.573CST",
      "lastUpdated" : "2016-11-09T17:01:22.614CST",
      "duration" : 247336,
      "sparkUser" : "spark",
      "completed" : true
      } ]
      }
    • Result analysis:

      By running this command, you can query information about a Spark application. Table 1 provides information about the application.

  • Obtaining information about the executor of a running application
    • Command for an alive executor:
      curl https://192.168.169.84:26001/proxy/application_1478570725074_0046/api/v1/applications/application_1478570725074_0046/executors?mode=monitoring --insecure
    • Commands for all alive and dead executors:
      curl https://192.168.169.84:26001/proxy/application_1478570725074_0046/api/v1/applications/application_1478570725074_0046/allexecutors?mode=monitoring --insecure

      192.168.195.232 is the service IP address of the active ResourceManager node, 26001 is the port number of ResourceManager, and application_1478570725074_0046 is the application ID in Yarn.

    • Command output:
      [{
      "id" : "driver",
      "hostPort" : "192.168.169.84:23886",
      "isActive" : true,
      "rddBlocks" : 0,
      "memoryUsed" : 0,
      "diskUsed" : 0,
      "activeTasks" : 0,
      "failedTasks" : 0,
      "completedTasks" : 0,
      "totalTasks" : 0,
      "totalDuration" : 0,
      "totalInputBytes" : 0,
      "totalShuffleRead" : 0,
      "totalShuffleWrite" : 0,
      "maxMemory" : 278019440,
      "executorLogs" : { }
      }, {
      "id" : "1",
      "hostPort" : "192.168.169.84:23902",
      "isActive" : true,
      "rddBlocks" : 0,
      "memoryUsed" : 0,
      "diskUsed" : 0,
      "activeTasks" : 0,
      "failedTasks" : 0,
      "completedTasks" : 0,
      "totalTasks" : 0,
      "totalDuration" : 0,
      "totalInputBytes" : 0,
      "totalShuffleRead" : 0,
      "totalShuffleWrite" : 0,
      "maxMemory" : 555755765,
      "executorLogs" : {
      "stdout" : "https://XTJ-224:26010/node/containerlogs/container_1478570725074_0049_01_000002/admin/stdout?start=-4096",
      "stderr" : "https://XTJ-224:26010/node/containerlogs/container_1478570725074_0049_01_000002/admin/stderr?start=-4096"
      }
      } ]
    • Result analysis:

      By running this command, you can query information about all executors (including drivers of the current application). Table 2 provides basic information about each executor.

      Table 2 Basic executor information

      Parameter

      Description

      id

      Executor ID

      hostPort

      IP address:port of the node of the executor

      executorLogs

      Executor log path

Enhanced REST APIs

  • SQL commands: Obtain all SQL statements and SQL statements that have the longest execution time.
    • SparkUI command:
      curl https://192.168.195.232:26001/proxy/application_1476947670799_0053/api/v1/applications/Spark-JDBCServerapplication_1476947670799_0053/SQL?mode=monitoring --insecure

      192.168.195.232 is the service IP address of the active ResourceManager node, 26001 is the port number of ResourceManager. application_1476947670799_0053 is the application ID in Yarn, and Spark-JDBCServer is the Spark application name.

    • JobHistory command:
      curl https://192.168.227.16:22500/api/v1/applications/application_1478570725074_0004-part1/SQL?mode=monitoring --insecure

      192.168.227.16 is the service IP address of the JobHistory node, and 22500 is the port number of the JobHistory node. application_1478570725074_0004-part1 is the application ID.

    • Command output:

      The query results of the SparkUI and JobHistory commands are as follows:

      {
      "longestDurationOfCompletedSQL" : [ {
      "id" : 0,
      "status" : "COMPLETED",
      "description" : "getCallSite at SQLExecution.scala:48",
      "submissionTime" : "2016/11/08 15:39:00",
      "duration" : "2 s",
      "runningJobs" : [ ],
      "successedJobs" : [ 0 ],
      "failedJobs" : [ ]
      } ],
      "sqls" : [ {
      "id" : 0,
      "status" : "COMPLETED",
      "description" : "getCallSite at SQLExecution.scala:48",
      "submissionTime" : "2016/11/08 15:39:00",
      "duration" : "2 s",
      "runningJobs" : [ ],
      "successedJobs" : [ 0 ],
      "failedJobs" : [ ]
      }]
      }
    • Result analysis:

      After running the commands, you can query information about all SQL statements of the current application (that is, the sqls part in the result) and information about the SQL statement with the longest execution time (that is, the longestDurationOfCompletedSQL part in the result). The following table describes information about each SQL statement.

      Table 3 Basic SQL statement information

      Parameter

      Description

      id

      ID of an SQL statement

      status

      Execution status of an SQL statement. The options are RUNNING, COMPLETED, and FAILED.

      runningJobs

      List of jobs that are being executed among the jobs generated by the SQL statement

      successedJobs

      List of jobs that are successfully executed among the jobs generated by the SQL statement

      failedJobs

      List of jobs that fail to be executed among the jobs generated by the SQL statement

  • JDBC/ODBC Server commands: Obtain the number of connections, the number of running SQL statements, as well as information about all sessions and all SQL statements.
    • Command:
      curl https://192.168.195.232:26001/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/sqlserver?mode=monitoring --insecure

      192.168.195.232 is the service IP address of the active ResourceManager node, 26001 is the port number of ResourceManager, and application_1476947670799_0053 is the application ID in Yarn.

    • Command output:
      {
      "sessionNum" : 1,
      "runningSqlNum" : 0,
      "sessions" : [ {
      "user" : "spark",
      "ip" : "192.168.169.84",
      "sessionId" : "9dfec575-48b4-4187-876a-71711d3d7a97",
      "startTime" : "2016/10/29 15:21:10",
      "finishTime" : "",
      "duration" : "1 minute 50 seconds",
      "totalExecute" : 1
      } ],
      "sqls" : [ {
      "user" : "spark",
      "jobId" : [ ],
      "groupId" : "e49ff81a-230f-4892-a209-a48abea2d969",
      "startTime" : "2016/10/29 15:21:13",
      "finishTime" : "2016/10/29 15:21:14",
      "duration" : "555 ms",
      "statement" : "show tables",
      "state" : "FINISHED",
      "detail" : "== Parsed Logical Plan ==\nShowTablesCommand None\n\n== Analyzed Logical Plan ==\ntableName: string, isTemporary: boolean\nShowTablesCommand None\n\n== Cached Logical Plan ==\nShowTablesCommand None\n\n== Optimized Logical Plan ==\nShowTablesCommand None\n\n== Physical Plan ==\nExecutedCommand ShowTablesCommand None\n\nCode Generation: true"
      } ]
      }
    • Result analysis:

      After running this command, you can query the number of sessions, the number of running SQL statements, as well as information about all sessions and SQL statements of the current JDBC/ODBC application. Table 4 provides information about each session. Table 5 provides information about each SQL statement.

      Table 4 Basic session information

      Parameter

      Description

      user

      User connected to the session

      ip

      IP address of the node where the session resides

      sessionId

      Session ID

      startTime

      Time when the session starts the connection

      finishTime

      Time when the session ends the connection

      duration

      Session connection duration

      totalExecute

      Number of SQL statements executed on the session

      Table 5 Basic SQL information

      Parameter

      Description

      user

      User who executes the SQL statement

      jobId

      List of job IDs contained in the SQL statement

      groupId

      ID of the group where the SQL statement resides

      startTime

      SQL start time

      finishTime

      SQL end time

      duration

      SQL execution duration

      statement

      SQL statement

      detail

      Logical plan and physical plan

  • Streaming commands: Obtain the average input frequency, average scheduling delay, average execution duration, and average total delay.
    • Command:
      curl https://192.168.195.232:26001/proxy/application_1477722033672_0008/api/v1/applications/NetworkWordCountapplication_1477722033672_0008/streaming?mode=monitoring --insecure

      192.168.195.232 is the service IP address of the active ResourceManager node, 26001 is the port number of ResourceManager. application_1477722033672_0008 is the application ID in Yarn, and NetworkWordCount is the Spark application name.

    • Command output:
      {
      "avgInputRate" : "0.00 events/sec",
      "avgSchedulingDelay" : "1 ms",
      "avgProcessingTime" : "72 ms",
      "avgTotalDelay" : "73 ms"
      }
    • Result analysis:

      After running this command, you can query the average input frequency, average scheduling delay, average execution duration, and average total delay of the current Streaming application.