Constraints on Hudi Tables

Hudi Table Type

• Copy On Write

  Copy-on-write (COW) tables store data in Parquet files. Internal update operations need to be performed by rewriting the original Parquet files.

  • Advantage: It is efficient because only one data file in the corresponding partition needs to be read.
  • Disadvantage: During data write, a previous copy needs to be copied and then a new data file is generated based on the previous copy. This process is time-consuming. Therefore, the data read by the read request lags behind.

• Merge On Read

  Merge-on-read (MOR) tables store data in a hybrid format combining columnar-based Parquet and row-based format Avro. Parquet files are used to store base data, and Avro files (also called log files) are used to store incremental data.

  • Advantage: Data is written to the delta log first, and the delta log size is small. Therefore, the write cost is low.
  • Disadvantage: Files need to be compacted periodically. Otherwise, there are a large number of fragment files. The read performance is poor because delta logs and old data files need to be merged.

  Table 1 Trade-off of two table types

  Trade-off	CopyOnWrite	MergeOnRead
  Data latency	High	Low
  Query latency	Low	High
  Update cost (I/O)	High (rewriting the entire Parquet file)	Low
  Parquet file size	Small (high update cost)	Large (low update cost)
  Write amplification	High	Low (depending on the compaction policy)

Notes and Constraints for Using Hudi Tables

• Hudi supports the use of Spark SQL in the DDL/DML syntax. However, when using DLI's metadata to submit SparkSQL jobs, some SQL syntax that directly operates on OBS paths is not supported. For details, refer to DLI Hudi SQL Syntax Reference.
• Writing Hudi tables and modifying Hudi table structures are not supported in HetuEngine. Only reading Hudi tables is supported.
• When creating a Hudi table, you must correctly configure primaryKey and preCombineField. Otherwise, there may be data inconsistency between the expected and actual results.
• When using the metadata service provided by DLI, creating DLI tables is not supported. Only OBS tables can be created, which means the table path must be configured through the LOCATION parameter.
• When using the metadata service provided by LakeFormation, both internal and external tables are supported. Note that when dropping an internal table, the data will also be deleted synchronously.
• When submitting Spark SQL or Flink SQL jobs, there is no need to manually configure the hoodie.write.lock.provider item for Hudi. However, when submitting Spark Jar jobs, you must manually the item. Refer to Hudi Lock Configuration for more details.
• The version mapping between Hudi and the queue compute engine is:

  Compute Engine	Version	Hudi Version
  Spark	3.3.1	0.11.0
  Flink	1.15	0.11.0
  HetuEngine	2.1.0	0.11.0

  To determine the compute engine versions supported by a queue, follow these steps: Log in to the DLI console and choose Resources > Queue Management in the navigation pane on the left. Locate the queue you want to query on the queue management page. Click the icon next to queue name to expand its details. Look for the supported versions in Supported Versions. For SQL queues, you cannot switch versions, so checking the default version will indicate the current compute engine version in use.