Help Center/ Data Lake Insight/ FAQs/ Problems Related to SQL Jobs/ O&M Guide/ Why Can't I Query Table Data After Data Is Imported to a DLI Partitioned Table Because the File to Be Imported Does Not Contain Data in the Partitioning Column?
Updated on 2024-01-23 GMT+08:00
View PDF
Share

Why Can't I Query Table Data After Data Is Imported to a DLI Partitioned Table Because the File to Be Imported Does Not Contain Data in the Partitioning Column?

Symptom

A CSV file is imported to a DLI partitioned table, but the imported file data does not contain the data in the partitioning column. The partitioning column needs to be specified for a partitioned table query. As a result, table data cannot be queried.

Possible Causes

When data is imported to a DLI partitionedtable, if the file data does not contain the partitioning column, the system specifies __HIVE_DEFAULT_PARTITION__ as the column by default. If a Spark job finds that the partition is empty, null is returned.

Solution

  1. Log in to the DLI management console. In the SQL editor, click Settings.
  2. Add spark.sql.forcePartitionPredicatesOnPartitionedTable.enabled and set it to false.
    Figure 1 Parameters
  3. Query the entire table or the partitioned table.
Parent topic: O&M Guide