Why Is No Data Queried in the DLI Table Created Using the OBS File Path When Data Is Written to OBS by a Flink Job Output Stream?

Symptom

After data is written to OBS through the Flink job output stream, data cannot be queried from the DLI table created in the OBS file path.

For example, use the following Flink result table to write data to the obs://obs-sink/car_infos path in OBS.

create sink stream car_infos_sink (
  carId string,
  carOwner string,
  average_speed double,
  buyday string
  ) partitioned by (buyday)
  with (
    type = "filesystem",
    file.path = "obs://obs-sink/car_infos",
    encode = "parquet",
    ak = "{{myAk}}",
    sk = "{{mySk}}"
);

create table car_infos (
  carId string,
  carOwner string,
  average_speed double
)
  partitioned by (buyday string)
  stored as parquet
  location 'obs://obs-sink/car_infos';

Solution

1. Check whether checkpointing is enabled for the Flink result table (car_infos_sink in the preceding example) when you create the job on DLI. If checkpointing is disabled, enable it and run the job again to generate OBS data files.

   To enable checkpointing, perform the following steps:

   1. Log in to the DLI management console. Choose Job Management > Flink Jobs in the left navigation pane. Locate the row that contains the target Flink job and click Edit in the Operation column.
   2. In the Running Parameters area, check whether Enable Checkpointing is enabled.
      Figure 1 Enable checkpointing
      
2. Check whether the structure of the Flink result table is the same as that of the DLI partitioned table. For the preceding example, check whether the fields of the car_infos_sink table are consistent with those of the car_infos table.
3. Check whether the partitioning information of the the partitioned table is restored after it is created using the OBS file. The following statement restore partitions of the car_infos table:

   alter table car_infos recover partitions;