Typical Hudi Configuration Parameters

hoodie.datasource.write.table.name

Name of the Hudi table to which data is written

None

hoodie.datasource.write.operation

Type of the operation for writing data to the Hudi table. Value options are as follows:

• upsert: updates and inserts data.
• delete: deletes data.
• insert: inserts data.
• bulk_insert: imports data during initial table creation. Do not use upsert or insert during initial table creation.
• insert_overwrite: performs insert and overwrite operations on static partitions.
• insert_overwrite_table: performs insert and overwrite operations on dynamic partitions. It does not immediately delete the entire table or overwrite the table. Instead, it overwrites the metadata of the Hudi table logically, and Hudi's cleaning mechanism removes obsolete data later. This operation is significantly faster than performing bulk_insert followed by overwrite.

upsert

hoodie.datasource.write.table.type

Type of the Hudi table. This parameter cannot be modified once the table is created. The value can be COPY_ON_WRITE or MERGE_ON_READ.

• COPY_ON_WRITE: indicates COW tables that store data in Parquet files. Update operations require rewriting the original Parquet files.
• MERGE_ON_READ: indicates MOR tables, which use a combination of columnar-based Parquet and row-based Avro formats to store data. Parquet files are used to store base data, while Avro files (also called log files) are used to store incremental data.

COPY_ON_WRITE

hoodie.datasource.write.precombine.field

Merges and deduplicates rows with the same key before write.

A specific table field

hoodie.datasource.write.payload.class

Class used to merge the records to be updated and the updated records during update. This parameter can be customized. You can compile it to implement your merge logic.

org.apache.hudi.common.model.DefaultHoodieRecordPayload

hoodie.datasource.write.recordkey.field

Unique primary key of the Hudi table

A specific table field

hoodie.datasource.write.partitionpath.field

Partition key. This parameter can be used together with hoodie.datasource.write.keygenerator.class to meet different partition needs.

None

hoodie.datasource.write.hive_style_partitioning

Flag to indicate whether to use Hive style partitioning. When this parameter is set to true, Hive partitions store data in separate directories based on partition field values, simplifying data management and querying. The recommended value is true.

• true: Hive style partitioning is used.
• false: Hive style partitioning is not used.

true

hoodie.datasource.write.keygenerator.class

Used with hoodie.datasource.write.partitionpath.field and hoodie.datasource.write.recordkey.field to generate the primary key and partition mode.

org.apache.hudi.keygen.ComplexKeyGenerator

hoodie.datasource.hive_sync.enable

Whether to enable Hive metadata synchronization when Hudi writes data. When this configuration is enabled, Hudi automatically synchronizes table metadata to the Hive Metastore during data writes. This allows Hive to query Hudi tables directly. Set this parameter to true to use Hive to centrally manage Hudi tables.

• true: Hive metadata synchronization is enabled.
• false: Hive metadata synchronization is not enabled.

false

hoodie.datasource.hive_sync.database

Name of the destination database to which the Hive table will be synchronized after Hive metadata synchronization is enabled.

default

hoodie.datasource.hive_sync.table

Name of the destination table to which the Hive table will be synchronized after Hive metadata synchronization is enabled. It is recommended that this value be the same as that of hoodie.datasource.write.table.name.

unknown

hoodie.datasource.hive_sync.username

Username for connecting to the Hive Metastore after Hive metadata synchronization is enabled.

hive

hoodie.datasource.hive_sync.password

Password for connecting to the Hive Metastore after Hive metadata synchronization is enabled.

hive

hoodie.datasource.hive_sync.jdbcurl

JDBC URL for connecting to the Hive Metastore after Hive metadata synchronization is enabled.

""

hoodie.datasource.hive_sync.use_jdbc

Whether to use Hive JDBC to connect to Hive and synchronize Hudi table information.

Set this parameter to false to invalidate the JDBC connection configuration.

• true: Hive JDBC is used to connect to Hive and synchronize Hudi table information.
• false: Hive JDBC is not used to connect to Hive and synchronize Hudi table information.

true

hoodie.datasource.hive_sync.partition_fields

Hive partition columns Partitioning significantly improves query performance, especially for large datasets, because queries only scan the relevant partition directories.

""

hoodie.datasource.hive_sync.partition_extractor_class

Class used to extract Hudi partition column values and convert them into Hive partition columns.

org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor

hoodie.datasource.hive_sync.support_timestamp

If the Hudi table contains a field of the timestamp type, set this parameter to true to synchronize the timestamp type to the Hive metadata.

The default value is false, indicating that the timestamp type is converted to bigint during synchronization by default. In this case, an error may occur when you query a Hudi table that contains a field of the timestamp type using SQL statements.

true

hoodie.index.class

Full path of a user-defined index, which must be a subclass of HoodieIndex. When this parameter is specified, the configuration takes precedence over that of hoodie.index.type.

""

hoodie.index.type

Index type. The default value is BLOOM. The possible options are BLOOM, HBASE, GLOBAL_BLOOM, SIMPLE, and GLOBAL_SIMPLE. The Bloom filter eliminates the dependency on an external system and is stored in the footer of a Parquet data file.

BLOOM

hoodie.index.bloom.num_entries

This is the number of entries to be stored in the bloom filter. Assume the maxParquetFileSize is 128 MB and averageRecordSize is 1024 bytes and hence a total of 130 KB records are recorded in a file. The default (60000) is roughly half of this approximation.

60000

hoodie.index.bloom.fpp

Error rate allowed given the number of entries. This is used to calculate how many bits should be assigned for the bloom filter and the number of hash functions. This is usually set very low (default: 0.000000001) to tradeoff disk space for lower false positives.

0.000000001

hoodie.bloom.index.parallelism

Parallelism for bloom filter index lookup, which involves a shuffle.

Increasing parallelism improves index-building performance but also increases resource consumption. By default, it is automatically calculated based on input workload characteristics.

0

hoodie.bloom.index.prune.by.ranges

Whether to optimize the index using range pruning when building a Bloom filter index. When this parameter is set to true, range information from files is used to speed up index lookup. This is particularly helpful if the keys have monotonously increasing prefixes, such as timestamp.

• true: Range pruning is enabled.
• false: Range pruning is disabled.

true

hoodie.bloom.index.use.caching

Whether to enable caching when building and using a Bloom filter index. When this parameter is set to true, the input RDD will be cached to speed up index lookup by reducing I/O for computing parallelism or affected partitions. This improves query performance, especially for frequently queried data.

• true: Caching is enabled.
• false: Caching is disabled.

true

hoodie.bloom.index.use.treebased.filter

Whether to enable interval tree-based file pruning optimization in the Bloom filter index. This mode speeds up file pruning based on key ranges when compared with the brute-force mode.

• true: Interval tree-based file pruning optimization is enabled.
• false: Interval tree-based file pruning optimization is disabled.

true

hoodie.bloom.index.bucketized.checking

Whether to enable bucketized bloom filtering in the Bloom filter index. Enabling bucketized bloom filtering reduces skew seen in sort based bloom index lookup.

• true: Bucketized bloom filtering is enabled.
• false: Bucketized bloom filtering is disabled.

true

hoodie.bloom.index.keys.per.bucket

This parameter is available only if bloomIndexBucketizedChecking is enabled and the index type is BLOOM.

This configuration controls the "bucket" size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading the Bloom filter to memory.

10000000

hoodie.bloom.index.update.partition.path

Whether to update the partition path during record updates when the index type is GLOBAL_BLOOM.

• true: An update including the partition path of a record that already exists will result in the insertion of the incoming record into the new partition and the deletion of the original record in the old partition.
• false: The original record will only be updated in the old partition.

true

hoodie.index.hbase.zkquorum

Mandatory. This parameter is available only when the index type is HBase. This parameter specifies the HBase ZooKeeper quorum URL to be connected.

None

hoodie.index.hbase.zkport

Mandatory. This parameter is available only when the index type is HBase. This parameter specifies the HBase ZooKeeper quorum port to be connected.

None

hoodie.index.hbase.zknode.path

Mandatory. This parameter is available only when the index type is HBase. It is the root znode that will contain all the znodes created and used by HBase.

None

hoodie.index.hbase.table

Mandatory. This parameter is available only when the index type is HBase. HBase table name to be used as an index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table.

None

hoodie.parquet.max.file.size

Specifies the target size for Parquet files generated in Hudi write phases. For DFS, this parameter needs to be aligned with the underlying file system block size for optimal performance.

120 * 1024 * 1024 byte

hoodie.parquet.block.size

Specifies the Parquet page size. Page is the unit of read in a Parquet file. In a block, pages are compressed separately.

120 * 1024 * 1024 byte

hoodie.parquet.compression.ratio

Specifies the expected compression ratio of Parquet data when Hudi attempts to adjust the size of a new Parquet file. If the size of the file generated by bulk_insert is smaller than the expected size, increase the value.

0.1

hoodie.parquet.compression.codec

Compression codec for parquet files. The default value is snappy. Possible options are gzip, snappy, uncompressed, and lzo.

• snappy: A fast compression algorithm suitable for most scenarios.
• gzip: Provides a high compression ratio, but compression and decompression speeds are slow.
• uncompressed: No compression is performed.
• lzo: Provides a high compression ratio but requires additional library support.

snappy

hoodie.logfile.max.size

Maximum size of a Hudi log file. It is the maximum size allowed for a log file before it is rolled over to the next version.

1GB

hoodie.logfile.data.block.max.size

Specifies the maximum size of a LogFile data block. It is the maximum size allowed for a single data block to be appended to a log file. It helps to ensure that the data appended to the log file is broken up into sizable blocks to prevent OOM errors. The size should be greater than the JVM memory.

256MB

hoodie.logfile.to.parquet.compression.ratio

Specifies the expected additional compression when records move from log files to Parquet files. It is used for MOR tables to send inserted content into log files and control the size of compacted Parquet files.

0.35

hoodie.clean.automatic

Whether data cleaning is performed automatically. Data cleaning is a critical Hudi function that deletes expired or unnecessary data files to optimize storage space and query performance.

• true: Hudi automatically performs data cleaning after each write operation.
• false: Data cleaning must be performed manually.

true

hoodie.cleaner.policy

Cleaning policy to manage the Hudi table. Hudi will delete the Parquet file of an old version to reclaim space. Any query or computation referring to this version of the file will fail. You are advised to ensure that the data retention time exceeds the maximum query execution time.

Available options are as follows:

• KEEP_LATEST_FILE_VERSIONS: keeps the most recent file versions and deletes older versions.
• KEEP_LATEST_COMMITS: keeps the most recent commits and deletes older ones.
• KEEP_LATEST_FILE_VERSIONS_BY_FILE_SIZE: keeps the most recent file versions while optimizing storage based on the file size.
• KEEP_LATEST_COMMITS_AND_FILE_VERSIONS: combines the KEEP_LATEST_COMMITS and KEEP_LATEST_FILE_VERSIONS policies to keep the most recent commits and file versions.

KEEP_LATEST_COMMITS

hoodie.cleaner.commits.retained

In Hudi, each write operation (such as an insert, update, or delete) generates a new commit. Each commit includes a unique timestamp that identifies the specific write operation.

This parameter specifies the number of commits to retain during Hudi cleaning operations. When the number of commits exceeds this value, Hudi deletes the oldest commits and their associated data files to reclaim storage space.

10

hoodie.keep.max.commits

In Hudi, each write operation generates a new commit. Each commit includes a unique timestamp that identifies the specific write operation.

Hudi supports time travel queries, which allow users to query the state of data as it existed at a specific point in time. To support this feature, Hudi retains data across multiple historical versions.

This parameter specifies the maximum number of commits that Hudi retains. When the number of commits exceeds this value, Hudi deletes the oldest commits and their associated data files to reclaim storage space.

30

hoodie.keep.min.commits

Minimum number of commits reserved for archiving operations.

20

hoodie.commits.archival.batch

Number of commit instants read in memory as a batch and archived together. Adjusting this parameter controls the frequency of archiving operations and the number of records processed in each batch.

10

hoodie.parquet.small.file.limit

The value must be smaller than that of maxFileSize. If maxFileSize is set to 0, this function is disabled. Small files always exist because of the large number of insert records in a partition of batch processing. Hudi provides an option to solve the problem of small files by masking inserts into this partition as updates to existing small files. The size here is the minimum file size that is considered as a "small file size".

104857600 B

hoodie.copyonwrite.insert.split.size

Parallelism for inserting and writing data. It is the number of inserts grouped for a single partition. Writing out 100 MB files with at least 1 KB records means 100 KB records exist in each file. Overprovision to 500 KB by default. To improve insert latency, adjust the value to match the number of records in a single file. If it is set to a smaller value, the file size will shrink (especially when compactionSmallFileSize is set to 0).

500000

hoodie.copyonwrite.insert.auto.split

Whether Hudi dynamically computes insertSplitSize based on the last 24 commit metadata. This function is disabled by default.

• true: Hudi dynamically calculates insertSplitSize based on metadata from the last 24 commits to automatically split large files and prevent excessive file growth.
• false: maintains file integrity during data insertion without performing automatic splitting.

true

hoodie.copyonwrite.record.size.estimate

Average record size. If specified, Hudi will use this parameter and not compute dynamically based on the last 24 commit metadata. There is no default value. This is critical in computing the insert parallelism and packing inserts into small files.

1024

hoodie.compact.inline

Whether to enable inline compression during write operations. Compaction is a crucial mechanism in Hudi for optimizing storage and performance. It improves read efficiency and reduces file fragmentation by merging small files and removing redundant data.

Inline Compaction: Compaction automatically performed during write operations (such as an insert, update, or delete). This approach eliminates the need for independent compaction tasks and improves overall write performance.

Async Compaction: Compaction performed in the background by independent tasks. This approach provides greater flexibility in controlling the frequency and timing of compaction, though it may increase system complexity.

• true: Compaction is triggered by the ingestion process during write operations.
• false: Compression is not performed during write operations.

true

hoodie.compact.inline.max.delta.commits

Maximum number of delta commits during inline compaction. This parameter is critical when inline compaction is enabled, as it determines the maximum number of delta commits merged within a single write operation.

5

hoodie.compaction.lazy.block.read

When CompactedLogScanner merges all log files, this parameter helps to choose whether the logblocks should be read lazily. Set it to true to use I/O-intensive lazy block read (low memory usage) or false to use memory-intensive immediate block read (high memory usage).

true

hoodie.compaction.reverse.log.read

HoodieLogFormatReader reads a log file in the forward direction from pos=0 to pos=file_length. If this parameter is set to true, Reader reads a log file in reverse direction from pos=file_length to pos=0. If this parameter is set to false, log files are read sequentially.

false

hoodie.cleaner.parallelism

The cleaning operation in Hudi removes obsolete data files, such as expired incremental files and redundant base files. Adjusting this parameter controls the degree of parallelism for the cleaning operation, balancing resource utilization against cleaning performance.

Increase this parameter if cleaning becomes slow.

200

hoodie.compaction.strategy

Which file groups are selected for compaction during each compaction run. By default, Hudi selects the log file with most accumulated unmerged data.

org.apache.hudi.table.action.compact.strategy.

LogFileSizeBasedCompactionStrategy

hoodie.compaction.target.io

Number of MBs to spend during compaction run for LogFileSizeBasedCompactionStrategy. This parameter can limit ingestion latency when compaction is run in inline mode.

500 * 1024 MB

hoodie.compaction.daybased.target.partitions

Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.

10

hoodie.compaction.payload.class

It needs to be same as class used during insert or upsert. Similar to writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file, and produce the final record to be written after compaction.

org.apache.hudi.common.model.DefaultHoodieRecordPayload

hoodie.schedule.compact.only.inline

Whether to only generate a compaction plan during inline compaction without executing it. This parameter is valid only when hoodie.compact.inline is set to true.

• true: only generates the compaction plan during inline compaction but defers execution to a separate process.
• false: allows both write operations and independent compaction tasks to generate compaction plans.

false

hoodie.run.compact.only.inline

Whether to execute only existing compaction plans when the run compaction SQL command is triggered. If no pre-existing plan is found, the command completes without taking further action.

• true: executes compaction exclusively during the write operation, disabling all independent asynchronous compaction tasks.
• false: allows compaction to be executed by either the write operation or independent compaction tasks.

false

hoodie.clustering.inline

Clustering optimizes data storage layout to increase query speed without reducing ingestion performance. Hudi clustering can be performed synchronously or asynchronously.

• true: enables synchronous clustering.
• false: disables synchronous clustering.

false

hoodie.clustering.inline.max.commits

Number of commits that trigger clustering during synchronous clustering.

This parameter limits the synchronous clustering scope and prevents excessive commits in a single write operation from reducing write performance.

4

hoodie.clustering.async.enabled

Whether to enable asynchronous clustering to reduce write latency and improve write performance.

• true: enables asynchronous clustering.
• false: disables asynchronous clustering.

false

hoodie.clustering.async.max.commits

Number of commits that trigger clustering during asynchronous clustering.

4

hoodie.clustering.plan.strategy.target.file.max.bytes

Maximum size of each file after clustering.

1024 * 1024 * 1024 byte

hoodie.clustering.plan.strategy.small.file.limit

Files smaller than the value of this parameter are clustered.

300 * 1024 * 1024 byte

hoodie.clustering.plan.strategy.sort.columns

Column used by clustering for sorting.

None

hoodie.layout.optimize.strategy

Clustering execution policy. The value can be linear, z-order, or hilbert.

linear

hoodie.layout.optimize.enable

This function must be enabled when z-order and hilbert are used.

false

hoodie.clustering.plan.strategy.class

Strategy class for filtering file groups for clustering. By default, files smaller than the value of hoodie.clustering.plan.strategy.small.file.limit are filtered.

org.apache.hudi.client.clustering.plan.strategy.SparkSizeBasedClusteringPlanStrategy

hoodie.clustering.execution.strategy.class

Strategy class for executing clustering (subclass of RunClusteringStrategy), which is used to define the execution mode of a cluster plan.

The default class sorts the file groups in the plan by the specified column and meets the configured target file size.

org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy

hoodie.clustering.plan.strategy.max.num.groups

Maximum number of file groups that can be selected during clustering. A larger value indicates a higher concurrency.

30

hoodie.clustering.plan.strategy.max.bytes.per.group

Maximum number of data records in each file group involved in clustering

2 * 1024 * 1024 * 1024 byte