Help Center/ MapReduce Service/ Component Development Specifications/ Hudi/ Spark on Hudi Stream Scenario Operation Guide/ Common Parameters and Application Scenarios
Updated on 2026-02-25 GMT+08:00
View PDF
Share

Common Parameters and Application Scenarios

Write

Parameter

Optional Value

Description

hoodie.sql.bulk.insert.enable

true or false

Whether to enable the bulkinsert write mode.

hoodie.combine.before.insert

true or false

Before the write operation, do not merge the data with the same primary key in the incremental data in case of data duplication.

hoodie.sql.insert.mode

upsert, strict, non-strict
  • upsert: The incremental data and the data with the same primary key in the inventory data are merged.
  • strict: The incremental data cannot contain the existing primary key in the inventory data. Otherwise, the incremental task fails.
  • non-strict: Data with the same primary key in incremental data and inventory data is not merged and is directly written.

hoodie.datasource.write.row.writer.enable

true or false

After this function is enabled, data is written using the native Parquet file writing mode of Spark, which provides better performance.

Common service scenarios: Initialize Hudi tables and migrate a large amount of data.

This initialization mode can be executed only once. It is used to migrate a large amount of data to the Hudi table for the first time and cannot be used for subsequent incremental tasks.

set hoodie.sql.bulk.insert.enable = true;
set hoodie.combine.before.insert = false;
set hoodie.sql.insert.mode = non-strict;
set hoodie.datasource.write.row.writer.enable = true;
insert into SinkTable select * from SourceTable;

Compaction

Parameter

Optional Value

Description

hoodie.compact.inline

The options include true and false.

Enable or disable compaction.

hoodie.schedule.compact.only.inline

The options include true and false.

After compaction is enabled, if this parameter is set to true, only compaction plans are generated.

hoodie.run.compact.only.inline

The options include true and false.

If this parameter is set to true after compaction is enabled, only existing compaction plans are executed.

hoodie.compact.inline.max.delta.commits

X (int value greater than 0)

After compaction is enabled, compaction is triggered only when the number of times write tasks are submitted exceeds X.

Clean

Parameter

Optional Value

Description

hoodie.clean.automatic

The options include true and false.

Enable or disable clean.

hoodie.clean.async

The options include true and false.

false: Disable the function.

hoodie.cleaner.commits.retained

X (int value greater than 0)

After clean is enabled, the data files corresponding to the write submission for the last X times are retained.

Archive

Parameter

Optional Value

Description

hoodie.archive.automatic

The options include true and false.

Enables or disables archive.

hoodie.archive.async

The options include true and false.

false: Disable the function.

hoodie.keep.min.commits

X (int value greater than 0)

After archive is enabled, the metadata corresponding to the write submissions of the last X times must be retained at least.

hoodie.keep.max.commits

Y (int value greater than 0)

After archive is enabled, the metadata corresponding to the write submissions of the last Y times must be retained at most.

O&M

The following three cases can be used to quickly operate and maintain all MOR and COW tables. Do not modify parameters. This O&M method is not suitable for long-term O&M. For details about the normal O&M process, see Selecting a Proper Table Service Execution Mode.

  • Common service scenario 1: Many compaction plans in the MOR table are not executed. You need to manually execute all compaction plans in the Hudi table immediately.

    This scenario does not affect the write task, provided that compaction plans exist in the Hudi table. Otherwise, this scenario does not take effect.

    set hoodie.clean.automatic = false;
set hoodie.clean.async = false;
set hoodie.archive.automatic = false;
set hoodie.archive.async= false;
set hoodie.compact.inline = true;
set hoodie.run.compact.only.inline = true;
run compaction on SinkTable;
  • Common service scenario 2: No compaction is performed on MOR tables, and no compaction plans are available in the table. In this case, you need to perform compaction, clean, and archive operations to improve performance.

    Before using this function, ensure that the write task has been stopped or no data is written by the write task.

    set hoodie.compact.inline = true;
set hoodie.compact.inline.max.delta.commits = 1;
set hoodie.clean.automatic = false;
set hoodie.clean.async = false;
set hoodie.cleaner.commits.retained = 2;
set hoodie.archive.automatic = false;
set hoodie.archive.async= false;
set hoodie.keep.min.commits = 3;
set hoodie.keep.max.commits = 4;
run compaction on <database name>. <table name>;
run clean on <database name>. <table name>;
run archivelog on <database name>.<table name>;
  • Common service scenario 3: COW table or MOR table + insert overwrite. If clean and archive are not performed, the number of data file versions and metadata is large, and a thorough cleanup is required.

    Before using this function, ensure that the write task has been stopped.

    set hoodie.cleaner.commits.retained = 1;
set hoodie.keep.min.commits = 2;
set hoodie.keep.max.commits = 3;
run clean on <database name>. <table name>;
run archivelog on <database name>.<table name>;