Configuring Dynamic Overwriting for Hive Table Partitions

Scenario

In earlier versions, when the insert overwrite syntax is used to overwrite partition tables, only partitions with specified expressions are matched, and partitions without specified expressions are deleted. In Spark2.3, partitions without specified expressions are automatically matched. The syntax is the same as that of the dynamic partition matching syntax of Hive.

Parameters

Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Configurations, click All Configurations, and search for the following parameter.

Parameter	Description	Default Value	Value Range
spark.sql.sources.partitionOverwriteMode	Specifies the mode for inserting data in partition tables by running the insert overwrite command, which can be STATIC or DYNAMIC. When it is set to STATIC, Spark deletes all partitions based on the matching conditions. When it is set to DYNAMIC, Spark matches partitions based on matching conditions and dynamically matches partitions without specified conditions.	STATIC	[STATIC,DYNAMIC]