Development Suggestions

Using Spark SQL to Create Tables

To ensure that Hudi tables can be read and written on both Spark and Flink, you are advised to use Spark SQL to create Hudi tables.

[Example] Use Spark SQL to create a Hudi table.

CREATE TABLE hudi_mor(
  id int,
  name string,
  age INT,
  `date` string
) using hudi OPTIONS(
type = 'mor',  
primaryKey = 'id',
precombineField = 'age',
hoodie.index.type = 'BUCKET',
hoodie.datasource.write.hive_style_partitioning = 'true',
hoodie.datasource.hive_sync.partition_extractor_class = 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
hoodie.bucket.index.num.buckets = '8'
)
location 'hdfs://hacluster/tmp/hudi_mor'
partitioned by (`date`);

Using a Spark Asynchronous Job to Compact a Hudi Table

Enabling compaction in Flink jobs consumes extra TaskManager resources, which affects the timeliness of importing Flink job data to the data lake. You are advised to disable compaction in Flink and instead enable Spark jobs to perform asynchronous compaction. For details, see Hudi Data Table Compaction Specifications.