Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Spark2x (for MRS 3.x or Later)/ Spark SQL Enterprise-Class Enhancements/ Filtering Partitions without Paths in Partitioned Tables
Updated on 2024-12-11 GMT+08:00
View PDF

Filtering Partitions without Paths in Partitioned Tables

Scenario

When you perform the select query in Hive partitioned tables, the FileNotFoundException exception is displayed if a specified partition path does not exist in HDFS. To avoid the preceding exception, configure spark.sql.hive.verifyPartitionPath parameter to filter partitions without paths.

Procedure

Perform either of the following methods to filter partitions without paths:

  • Configure the following parameter in the spark-defaults.conf file on Spark client.
    Table 1 Parameter description

    Parameter

    Description

    Default Value

    spark.sql.hive.verifyPartitionPath

    Whether to filter partitions without paths when reading Hive partitioned tables.

    true: enables the filtering

    false: disables the filtering

    false
  • When running the spark-submit command to submit an application, configure the --conf parameter to filter partitions without paths.
    For example: 
    spark-submit --class org.apache.spark.examples.SparkPi  --conf spark.sql.hive.verifyPartitionPath=true $SPARK_HOME/lib/spark-examples_*.jar