Optimizing Datasource Tables

Updated on 2022-02-22 GMT+08:00

View PDF

Save the partition information about the datasource table to the Metastore and process partition information in the Metastore.

Optimize the datasource tables, support syntax such as adding, deletion, and modification in the table based on partitions, improving compatibility with Hive.
Support statements of partition tailoring and push down to the Metastore to filter unmatched partitions.
Example:
```
select count(*) from table where partCol=1;    //partCol (partition column)
```
You need only to process data corresponding to partCol=1 when performing the TableScan operation in the physical plan.

If you want to enable Datasource table optimization, configure the spark-defaults.conf file on the Spark client.

**Table 1** Parameter description
Parameter	Description	Default Value
spark.sql.hive.manageFilesourcePartitions	Specifies whether to enable Metastore partition management (including datasource tables and converted Hive). true indicates enabling Metastore partition management. In this case, datasource tables are stored in Hive and Metastore is used to tailor partitions in query statements. false indicates disabling Metastore partition management.	true
spark.sql.hive.metastorePartitionPruning	Specifies whether to support pushing down predicate to Hive Metastore. true indicates supporting pushing down predicate to Hive Metastore. Only the predicate of Hive tables is supported. false indicates not supporting pushing down predicate to Hive Metastore.	true
spark.sql.hive.filesourcePartitionFileCacheSize	The cache size of the partition file metadata in the memory. All tables share a cache that can use up to specified num bytes for file metadata. This parameter is valid only when spark.sql.hive.manageFilesourcePartitions is set to true.	250 * 1024 * 1024
spark.sql.hive.convertMetastoreOrc	The processing approach of ORC tables. false: Spark SQL uses Hive SerDe to process ORC tables. true: Spark SQL uses the Spark built-in mechanism to process ORC tables.	true

Parent topic: Spark SQL and DataFrame Tuning

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.