Configuring Streaming Reading of Spark Driver Execution Results

Updated on 2024-10-08 GMT+08:00

View PDF

Scenario

When a query statement is executed, the returned result may be large (containing more than 100,000 records). In this case, JDBCServer out of memory (OOM) may occur. Therefore, the data aggregation function is provided to avoid OOM without sacrificing the performance.

Configuration

Two data aggregation function configuration parameters are provided. The two parameters are set in the tunning option on the Spark JDBCServer server. After the setting is complete, restart JDBCServer.

**Table 1** Parameter description
Parameter	Description	Default Value
spark.sql.bigdata.thriftServer.useHdfsCollect	Indicates whether to save result data to HDFS instead of the memory. Advantages: The query result is stored in HDFS. Therefore, JDBCServer OOM does not occur. Disadvantages: The query is slow. true: Result data is saved to HDFS. false: This function is disabled. NOTICE: When spark.sql.bigdata.thriftServer.useHdfsCollect is set to true, result data is saved to HDFS. However, the job description on the native JobHistory page cannot be associated with the corresponding SQL statement. In addition, the execution ID in the spark-beeline command output is null. To solve the JDBCServer OOM problem and ensure correct information display, you are advised to set spark.sql.userlocalFileCollect.	false
spark.sql.uselocalFileCollect	Indicates whether to save result data to the local disk instead of memory. Advantages: In the case of small data volume, the performance loss can be ignored compared with the data storage mode using the native memory. In the case of large data volume (hundreds of millions of data records), the performance is much better than that when data is stored in the HDFS and native memory. Disadvantages: Optimization is required. In the case of large data volume, it is recommended that the JDBCServer driver memory be 10 GB and each core of the executor be allocated with 3 GB memory. true: This function is enabled. false: This function is disabled.	false
spark.sql.collect.Hive	This parameter is valid only when spark.sql.uselocalFileCollect is set to true. It indicates whether to save the result data to a disk in direct serialization mode or in indirect serialization mode. Advantage: For queries of tables with a large number of partitions, the aggregation performance of the query results is better than that of the storage mode that query results are directly stored on the disk. Disadvantages: The disadvantages are the same as those when spark.sql.uselocalFileCollect is enabled. true: This function is enabled. false: This function is disabled.	false
spark.sql.collect.serialize	This parameter takes effect only when both spark.sql.uselocalFileCollect and spark.sql.collect.Hive are set to true. The function is to further improve performance. java: Data is collected in Java serialization mode. kryo: Data is collected in kryo serialization mode. The performance is better than that when the Java serialization mode is used.	java

NOTE:

spark.sql.bigdata.thriftServer.useHdfsCollect and spark.sql.uselocalFileCollect cannot be set to true at the same time.

Parent topic: Spark Core Enterprise-Class Enhancements

Previous topic: Using the Spark Small File Combination Tool

Next topic: Spark SQL Enterprise-Class Enhancements

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot