To Hudi

Basic parameters

Database Name

Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed.

dbadmin

Table Name

Click the icon next to the text box. The dialog box for selecting the table is displayed.

This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. For details, see Incremental Synchronization Using the Macro Variables of Date and Time.

cdm

Table Preparation Mode

Whether to automatically create Hudi tables

• One-click creation: The destination table is automatically created.
• Auto creation: If the destination database does not contain the table specified by Table Name, CDM will automatically create the table. If the table specified by Table Name already exists, no table is created and data is written to the existing table.

Auto creation

Write Mode

Data write mode

• TRUNCATE+LOAD: The TRUNCATE statement is executed to clear data in partitions before new data is written.
• LOAD: No operation is performed before data is written.
• INSERT_OVERWRITE: Data is overwritten.

LOAD

Partition

This parameter is displayed when Write Mode is TRUNCATE+LOAD or INSERT_OVERWRITE.

Partition information. To write data to a partitioned table, you can select the partitions to write data to.

year=2020,location=sun

Run Queue

Value of the request parameter --queue, which determines the queue to which a Spark task is submitted for execution.

default

Advanced attributes

SQL Execution Parameters

When writing data to Hudi, you can configure Spark parameters. The parameters you enter here will be added to the request body as --conf parameters. Spark driver and executor parameters are also supported.

--driver-memory=2G,--executor-memory=2G,--driver-cores=1,--executor-cores=1,--num-executors=2

DB Write Time Field

When a table is automatically created, this field is automatically added to the table creation statement. When the data is written to the Hudi table, the value of this field is the current time. The field must be of the timestamp type.

N/A

Write Parameters

Parameter configured using the set syntax to control the insertion of data into Hudi through a Spark SQL statement

• hoodie.datasource.write.operation: type of the operation for writing data into a Hudi table, which determines how data is written and merged. The default value is insert.
  NOTE:

  Optional values of the hoodie.datasource.write.operation parameter are as follows:

  • upsert (recommended): incremental data synchronization (CDC)

    If there is a record key, data is updated. If there is no record key, data is inserted. This is the most commonly used operation mode and applies to scenarios such as CDC synchronization and incremental data merging.

  • insert: batch insertion without duplicate data

    Only insertion is performed, and no update is performed. If there is a record key, the behavior depends on the hoodie.insert.dup.policy configuration (the default value is FAIL, indicating that an error is reported). This mode applies to scenarios where there is no duplicate data. It provides better performance than upsert.

  • bulk_insert: initial full data loading

    Data is written in batches directly. The Hudi index mechanism is bypassed. This mode is suitable for initial full data loading. It provides the fastest speed, but may not eliminate duplicate data.

  • delete: deletion of specified records

    Existing records are deleted based on the record key. The data to be written only needs to contain the record key field.

  • delete_partitions: deletion of data in a specified partition

    All records in a specified partition are deleted. You do not need to specify a record key.

  • insert_overwrite: overwrite of all data in a specified partition

    The data in a specified partition is replaced with new data (the partition is cleared before new data is written). This mode is applicable to scenarios where all data in a partition needs to be overwritten.

  • insert_overwrite_table: full table recreation

    Existing data in a table is cleared before new data is written to it.

• hoodie.combine.before.upsert: whether to precombine input data before an upsert operation. The default value is true.
  NOTE:

  Optional values of the hoodie.combine.before.upsert parameter are as follows:

  • true (recommended): The data source may contain duplicate record keys in the same batch.

    Before performing an upsert operation, Hudi combines (deduplicates) records with the same record key in the input batch and retains only the record with the largest sorting value.

  • false: The data source contains a unique record key in a single batch, reducing the precombine overhead.

    The precombine step is skipped, and all records are forwarded to the upsert process.

hoodie.combine.before.upsert: true

Compaction Mode

Compaction execution mode. This parameter takes effect only for MOR tables.

• Asynchronous compaction: After data is written to Hudi, only a compaction execution plan is generated. Clean and archive operations are not performed. This mode must be used together with asynchronous compaction scripts.
• Synchronous compaction: The compaction, clean, and archive operations are performed immediately after data is written to the Hudi table, which increases the job execution time. By default, five commits trigger compaction. By default, at least 20 deltacommits are retained, and a maximum of 30 deltacommits are retained. By default, 10 data files corresponding to deltacommits are retained after clean operations.

Asynchronous compaction