Spark ThriftServer APIs

Updated on 2024-12-10 GMT+08:00

View PDF

Introduction

ThriftServer is another HiveServer2 implementation in Hive. It uses the Spark structured query language (SQL) to process the SQL statements, providing higher performance than Hive.

ThriftServer is a JDBC API. Users can use JDBC to connect to Thrift Server to access SparkSQL data. When ThriftServer is started, a SparkSQL application is started, and the clients connected through the JDBC share the resources of the SparkSQL application, that is, different users can share data. When ThriftServer is started, a monitor is enabled to wait for the connections and queries submitted by the JDBC client. Therefore, when configuring ThriftServer, you must configure at least the host name and port number of ThriftServer. If you want to use Hive data, you also need to provide the URIs of Hive Metastore.

ThriftServer starts a JDBC service on port 10000 of the installation node by default. Users can connect to ThriftServer using Beeline or running the JDBC client code to run SQL statements.

For other information about ThriftServer, visit Spark's official website.

Beeline

For details about the Beeline connection modes provided by the open source community, visit https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients.

JDBC Client Code

You can run the JDBC client code to connect to the ThriftServer to access SparkSQL data.

Enhanced Features

Compared with the open source community, MRS provides two enhanced features: ThriftServer HA solution and timeout interval setting for ThriftServer connections.

In the ThriftServer HA solution, when the active ThriftServer node is faulty, the standby node can automatically switch to the active one to provide services for the cluster. The operations of using Beeline or JDBC client code for connection are the same.

The difference between the character strings used for connecting to ThriftServer in HA or non-HA modes is that you need to replace ip:port with ha-cluster for connecting to ThriftServer in HA mode. Table 1 provides other parameters that will be used.

**Table 1** List of client parameters
Parameter	Description	Default Value
spark.thriftserver.ha.enabled	Indicates whether to enable the HA mode. The value true indicates that the HA mode is enabled. If the HA mode is enabled, change host:port to ha-cluster in the connection string. Otherwise, the system automatically exits the HA mode.	false
spark.thriftserver.zookeeper.dir	ThriftServer path for storing metadata on the ZooKeeper. The parameter value must be the same as those on the server. The subdirectory named active_thriftserver in this directory is used for storing the IP address and port number of the Hive ThriftServer.	/thriftserver
spark.deploy.zookeeper.url	ZooKeeper URL. The parameter value must be the same as those on the server.	-
spark.thriftserver.retry.times	Maximum number of attempts to connect to the server. If the value is set to a negative number or zero, the client does not attempt to connect to the server again.	5
spark.thriftserver.retry.wait.time	Interval for attempting to reconnect to the server. The unit is second.	10

The parameters in Table 1 must be configured in the hive-site.xml file in classpath on the client. For example:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
    <property>
        <name>spark.thriftserver.ha.enabled</name>
        <value>true</value>
    </property>
</configuration>

The spark.deploy.zookeeper.url parameter can also be replaced with zk.quorum in the connection string. For example:

!connect jdbc:hive2://ha-cluster/default;zk.quorum=spark25:2181,spark
26:2181,spark27:2181

Set a timeout interval for a connection between the client and the ThriftServer.
- Beeline
  In the case of network congestion, this feature prevents the beeline from being suspended due to the infinite waiting for the response of the server. The configuration method is as follows:
  
  When the beeline is started, append --socketTimeOut=n, where n indicates the timeout interval (unit: second) for waiting for the service response. The default value is 0, indicating that the beeline never times out. You are advised to set this parameter to the maximum value allowed based on the service scenario.
- JDBC client code
  In the case of network congestion, this feature prevents the client from being suspended due to the infinite waiting for the response of the server. The configuration method is as follows:
  
  Before using the DriverManager.getConnection method to obtain the JDBC connection, add the DriverManager.setLoginTimeout(n) method to configure a timeout interval. n indicates the timeout interval for waiting for the return result from the server. The unit is second, the type is Int, and the default value is 0 (indicating never timing out). You are advised to set this parameter to the maximum value allowed based on the service scenario.