Updated on 2024-10-09 GMT+08:00

Configuring the Drop Partition Command to Support Batch Deletion

This section applies only to MRS 3.2.0 or later.

Scenario

Currently, the Drop Partition command in Spark supports partition deletion using only the equal sign (=). This configuration allows multiple filter criteria to be used to delete partitions in batches, for example, <, <=, >, >=, !>, and !<.

Configuration

Log in to FusionInsight Manager, choose Cluster > Services > Spark2x > Configurations, click All Configurations, and search for the following parameters.

Parameter

Description

Default Value

spark.sql.dropPartitionsInBatch.enabled

If this parameter is set to true, the Drop Partition command supports the following filter criteria: <, <=, >, >=, !>, and !<.

true

spark.sql.dropPartitionsInBatch.limit

Indicates the maximum number of partitions that can be batch dropped.

1000