Spark Scenario

MOR Table Parameter Simplification

create table hudi_mor(Service field) using hudi options(
type="mor",
primaryKey="Primary key field",
hoodie.bucket.index.num.buckets="20"
)

The MOR table requires only the three preceding parameters. The following parameters are automatically added to set the preCombineField attribute to the primary key field (using the first field for composite keys), set the index to BUCKET, set the write mode to LSM, set the Parquet file compression format to zstd, and enable the log index. These settings improve read/write performance and optimize storage. Note that the MOR table uses the BUCKET index by default. You must manually set hoodie.bucket.index.num.buckets to specify the number of buckets. The preceding table creation statements achieve an equivalent effect as follows:

create table hudi_mor(Service field) using hudi options(
type="mor",
primaryKey="Primary key field",
preCombineField="Primary key field",
hoodie.index.type="BUCKET",
hoodie.bucket.index.num.buckets="20",
hoodie.lsm.style="true",
hoodie.parquet.compression.codec="zstd",
hoodie.log.index.enable="true",
hoodie.datasource.write.hive_style_partitioning="true"
)

COW Table Parameter Simplification

create table hudi_cow(Service field) using hudi options(
primaryKey="Primary key field"
)

Bulk Insert (Only Writing New Data Without Updating Existing Data)

To enable INSERT INTO and INSERT OVERWRITE, you only need to add set hoodie.datasource.write.operation=bulk_insert before the INSERT statement. The effect is equivalent to the following:

set hoodie.datasource.write.operation=bulk_insert;
set hoodie.bucket.support.bulk.insert=true;
set hoodie.combine.before.insert=true;
set hoodie.bulkinsert.sort.mode=NONE;
insert into ......

Table Management

If the MRS cluster has the LDMS service, you can add the hoodie.managed.by.ldms="true" parameter when creating MOR and COW tables. The LDMS service then manages the compaction, clean, and archive operations of these tables. Note that for MOR tables, the write task should only generate a compaction plan and disable clean and archive operations. For COW tables, the write task should disable clean and archive operations.