Broaden Support for Hive Partition Pruning Predicate Pushdown

Scenario

In earlier versions, the predicate for pruning Hive table partitions is pushed down. Only comparison expressions between column names and integers or character strings can be pushed down. In version 2.3, pushdown of the null, in, and, or expressions are supported. In version 3.1.1 or later, comparison expressions between the column name and the char or varchar type, including !=, like, not like, and not in, can be pushed down. Currently, only % can be used as the wildcard for like and not like.

Formats supported by like pushdown expressions of the char type: {value}% and %{value}%
Formats supported by not like pushdown expressions: {value}% and %{value}%
Formats supported by like pushdown expressions of the varchar type: {value}%, %{value}%, %{value}, and {value1}%{value2}.
Formats supported by not like pushdown expressions: {value}%, %{value}%, %{value}, and {value1}%{value2}.

Parameters

Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations, and search for the following parameters:

Parameter	Description	Default Value	Value Range
spark.sql.hive.advancedPartitionPredicatePushdown.enabled	Specifies whether to broaden the support for Hive partition pruning predicate pushdown.	true	[true,false]
spark.sql.hive.varcharPartitionPredicatePushdown.enabled	Whether to support the pushdown of predicates of the char and varchar types.	false	[true,false]