Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Spark/Spark2x/ Spark SQL Enterprise-Class Enhancements/ Configuring Dynamic Overwriting for Hive Table Partitions
Updated on 2024-10-09 GMT+08:00
View PDF
Share

Configuring Dynamic Overwriting for Hive Table Partitions

Scenario

In earlier versions, when the insert overwrite syntax is used to overwrite partition tables, only partitions with specified expressions are matched, and partitions without specified expressions are deleted. In Spark2.3, partitions without specified expressions are automatically matched. The syntax is the same as that of the dynamic partition matching syntax of Hive.

Parameters

Log in to FusionInsight Manager, choose Cluster > Services > Spark2x > Configurations, click All Configurations, and search for the following parameters.

Parameter

Description

Default Value

Value Range

spark.sql.sources.partitionOverwriteMode

Specifies the mode for inserting data in partition tables by running the insert overwrite command, which can be STATIC or DYNAMIC.

  • When it is set to STATIC, Spark deletes all partitions based on the matching conditions.
  • When it is set to DYNAMIC, Spark matches partitions based on matching conditions and dynamically matches partitions without specified conditions.

STATIC

[STATIC,DYNAMIC]