Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ FAQs About Spark Application Development/ Spark APIs/ Spark ThriftServer APIs
Updated on 2024-08-16 GMT+08:00
View PDF
Share

Spark ThriftServer APIs

Introduction

ThriftServer is another HiveServer2 implementation in Hive. It uses the Spark structured query language (SQL) to process the SQL statements, providing higher performance than Hive.

ThriftServer is a JDBC API. Users can use JDBC to connect to Thrift Server to access SparkSQL data. When ThriftServer is started, a SparkSQL application is started, and the clients connected through the JDBC share the resources of the SparkSQL application, that is, different users can share data. When ThriftServer is started, a monitor is enabled to wait for the connections and queries submitted by the JDBC client. Therefore, when configuring ThriftServer, you must configure at least the host name and port number of ThriftServer. If you want to use Hive data, you also need to provide the URIs of Hive Metastore.

ThriftServer starts a JDBC service on port 10000 of the installation node by default. Users can connect to ThriftServer using Beeline or running the JDBC client code to run SQL statements.

For more information about ThriftServer, visit the Spark official website at http://spark.apache.org/docs/1.5.1/sql-programming-guide.html#distributed-sql-engine.

Beeline

For details about the Beeline connection modes provided by the open source community, visit https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients.

JDBC Client Code

You can run the JDBC client code to connect to the ThriftServer to access SparkSQL data.

Enhanced Features

Compared with the open source community, MRS provides two enhanced features: ThriftServer HA solution and timeout interval setting for ThriftServer connections.

  • In the ThriftServer HA solution, when the active ThriftServer node is faulty, the standby node can automatically switch to the active one to provide services for the cluster. The operations of using Beeline or JDBC client code for connection are the same.

    The difference between the character strings used for connecting to ThriftServer in HA or non-HA modes is that you need to replace ip:port with ha-cluster for connecting to ThriftServer in HA mode. Table 1 provides other parameters that will be used.

    Table 1 List of client parameters

    Parameter

    Description

    Default Value

    spark.thriftserver.ha.enabled

    Indicates whether to enable the HA mode. The value true indicates that the HA mode is enabled. If the HA mode is enabled, change host:port to ha-cluster in the connection string. Otherwise, the system automatically exits the HA mode.

    false

    spark.thriftserver.zookeeper.dir

    ThriftServer path for storing metadata on the ZooKeeper. The parameter value must be the same as those on the server. The subdirectory named active_thriftserver in this directory is used for storing the IP address and port number of the Hive ThriftServer.

    /thriftserver

    spark.deploy.zookeeper.url

    ZooKeeper URL. The parameter value must be the same as those on the server.

    -

    spark.thriftserver.retry.times

    Maximum number of attempts to connect to the server. If the value is set to a negative number or zero, the client does not attempt to connect to the server again.

    5

    spark.thriftserver.retry.wait.time

    Interval for attempting to reconnect to the server. The unit is second.

    10

    The parameters in Table 1 must be configured in the hive-site.xml file in classpath on the client. For example:

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
    <property>
        <name>spark.thriftserver.ha.enabled</name>
        <value>true</value>
    </property>
</configuration>

    The spark.deploy.zookeeper.url parameter can also be replaced with zk.quorum in the connection string. For example:

    !connect jdbc:hive2://ha-cluster/default;zk.quorum=spark25:2181,spark
26:2181,spark27:2181
  • Set a timeout interval for a connection between the client and the ThriftServer.
    • Beeline

      In the case of network congestion, this feature prevents the beeline from being suspended due to the infinite waiting for the response of the server. The configuration method is as follows:

      When the beeline is started, append --socketTimeOut=n, where n indicates the timeout interval (unit: second) for waiting for the service response. The default value is 0, indicating that the beeline never times out. You are advised to set this parameter to the maximum value allowed based on the service scenario.

    • JDBC client code

      In the case of network congestion, this feature prevents the client from being suspended due to the infinite waiting for the response of the server. The configuration method is as follows:

      Before using the DriverManager.getConnection method to obtain the JDBC connection, add the DriverManager.setLoginTimeout(n) method to configure a timeout interval. n indicates the timeout interval for waiting for the return result from the server. The unit is second, the type is Int, and the default value is 0 (indicating never timing out). You are advised to set this parameter to the maximum value allowed based on the service scenario.

Parent topic: Spark APIs