Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Spark/Spark2x/ Spark Core Enterprise-Class Enhancements/ Configuring Streaming Reading of Spark Driver Execution Results
Updated on 2024-12-13 GMT+08:00
View PDF
Share

Configuring Streaming Reading of Spark Driver Execution Results

Scenario

When a query statement is executed, the returned result may be large (containing more than 100,000 records). In this case, JDBCServer out of memory (OOM) may occur. Therefore, the data aggregation function is provided to avoid OOM without sacrificing the performance.

Configuration

Two data aggregation function configuration parameters are provided. The two parameters are set in the tunning option on the Spark JDBCServer server. After the setting is complete, restart JDBCServer.

Table 1 Parameter description

Parameter

Description

Default Value

spark.sql.bigdata.thriftServer.useHdfsCollect

Indicates whether to save result data to HDFS instead of the memory.

Advantages: The query result is stored in HDFS. Therefore, JDBCServer OOM does not occur.

Disadvantages: The query is slow.

  • true: Result data is saved to HDFS.
  • false: This function is disabled.
    NOTICE:

    When spark.sql.bigdata.thriftServer.useHdfsCollect is set to true, result data is saved to HDFS. However, the job description on the native JobHistory page cannot be associated with the corresponding SQL statement. In addition, the execution ID in the spark-beeline command output is null. To solve the JDBCServer OOM problem and ensure correct information display, you are advised to set spark.sql.userlocalFileCollect.

false

spark.sql.uselocalFileCollect

Indicates whether to save result data to the local disk instead of memory.

Advantages: In the case of small data volume, the performance loss can be ignored compared with the data storage mode using the native memory. In the case of large data volume (hundreds of millions of data records), the performance is much better than that when data is stored in the HDFS and native memory.

Disadvantages: Optimization is required. In the case of large data volume, it is recommended that the JDBCServer driver memory be 10 GB and each core of the executor be allocated with 3 GB memory.

  • true: This function is enabled.
  • false: This function is disabled.

false

spark.sql.collect.Hive

This parameter is valid only when spark.sql.uselocalFileCollect is set to true. It indicates whether to save the result data to a disk in direct serialization mode or in indirect serialization mode.

Advantage: For queries of tables with a large number of partitions, the aggregation performance of the query results is better than that of the storage mode that query results are directly stored on the disk.

Disadvantages: The disadvantages are the same as those when spark.sql.uselocalFileCollect is enabled.

  • true: This function is enabled.
  • false: This function is disabled.

false

spark.sql.collect.serialize

This parameter takes effect only when both spark.sql.uselocalFileCollect and spark.sql.collect.Hive are set to true.

The function is to further improve performance.

  • java: Data is collected in Java serialization mode.
  • kryo: Data is collected in kryo serialization mode. The performance is better than that when the Java serialization mode is used.

java

spark.sql.bigdata.thriftServer.useHdfsCollect and spark.sql.uselocalFileCollect cannot be set to true at the same time.