Configuring Streaming Reading of Driver Execution Results
Scenario
When a query statement is executed, the returned result may be large (containing more than 100,000 records). In this case, JDBCServer out of memory (OOM) may occur. Therefore, the data aggregation function is provided to avoid OOM without sacrificing the performance.
Configuration
Two data aggregation function configuration parameters are provided. The two parameters are set in the tunning option on the Spark JDBCServer server. After the setting is complete, restart JDBCServer.
Parameter |
Description |
Default Value |
---|---|---|
spark.sql.bigdata.thriftServer.useHdfsCollect |
Indicates whether to save result data to HDFS instead of the memory. Advantages: The query result is stored in HDFS. Therefore, JDBCServer OOM does not occur. Disadvantages: The query is slow.
|
false |
spark.sql.uselocalFileCollect |
Indicates whether to save result data to the local disk instead of memory. Advantages: In the case of small data volume, the performance loss can be ignored compared with the data storage mode using the native memory. In the case of large data volume (hundreds of millions of data records), the performance is much better than that when data is stored in the HDFS and native memory. Disadvantages: Optimization is required. In the case of large data volume, it is recommended that the JDBCServer driver memory be 10 GB and each core of the executor be allocated with 3 GB memory.
|
false |
spark.sql.collect.Hive |
This parameter is valid only when spark.sql.uselocalFileCollect is set to true. It indicates whether to save the result data to a disk in direct serialization mode or in indirect serialization mode. Advantage: For queries of tables with a large number of partitions, the aggregation performance of the query results is better than that of the storage mode that query results are directly stored on the disk. Disadvantages: The disadvantages are the same as those when spark.sql.uselocalFileCollect is enabled.
|
false |
spark.sql.collect.serialize |
This parameter takes effect only when both spark.sql.uselocalFileCollect and spark.sql.collect.Hive are set to true. The function is to further improve performance.
|
java |
spark.sql.bigdata.thriftServer.useHdfsCollect and spark.sql.uselocalFileCollect cannot be set to true at the same time.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.