What Should I Do If New Fields Cannot Be Found Using the DLI SQL Statement show create table When Synchronization of New Fields Is Enabled for a DLI-to-Hudi Real-Time Migration Job?

Symptom

Add Column is set to Normal processing in DDL Message Real-Time Synchronization Rule. When field f1 is added to the source, the field can be found using select * (Figure 2), but cannot be found using show create table (Figure 3).

Figure 1 DDL configuration

Figure 2 Querying fields

Figure 3 Querying the table structure

Possible Causes

DataArts Migration cannot synchronize the metadata information of the new fields to the DLI catalog. You need to configure a DLI SQL command to periodically execute the metadata synchronization task.

Solution

You need to manually synchronize Hive metadata by running the following DLI SQL command:

call sync_hive(table => 'db_name.table_name', tablePath => 'obs://xxxxx')

For details about the syntax, see SYNC_HIVE.

You are advised to configure the metadata synchronization command in the compaction script to periodically synchronize metadata.

For details about how to configure a periodic job, see How Do I Configure a Spark Periodic Task for Hudi Compaction?.