Help Center/ DataArts Studio/ User Guide/ DataArts Migration (Offline Jobs)/ Configuring Destination Job Parameters/ To Hudi
Updated on 2026-08-13 GMT+08:00
View PDF
Share

To Hudi

MRS Spark JDBC Server cannot intercept submitted SQL statements. If a Hudi connection uses the JDBC mode, SQL statements submitted after the job is stopped may fail to be canceled.

Table 1 Parameter description

Type

Parameter

Description

Recommended Configuration

Basic parameters

Database Name

Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed.

dbadmin

Table Name

Click the icon next to the text box. The dialog box for selecting the table is displayed.

This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. For details, see Incremental Synchronization Using the Macro Variables of Date and Time.

NOTE:

If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).

cdm

Table Preparation Mode

Whether to automatically create Hudi tables

  • One-click creation: The destination table is automatically created.
  • Auto creation: If the destination database does not contain the table specified by Table Name, CDM will automatically create the table. If the table specified by Table Name already exists, no table is created and data is written to the existing table.

Auto creation

Write Mode

Data write mode

  • TRUNCATE+LOAD: The TRUNCATE statement is executed to clear data in partitions before new data is written.
  • LOAD: No operation is performed before data is written.
  • INSERT_OVERWRITE: Data is overwritten.

LOAD

Partition

This parameter is displayed when Write Mode is TRUNCATE+LOAD or INSERT_OVERWRITE.

Partition information. To write data to a partitioned table, you can select the partitions to write data to.

year=2020,location=sun

Run Queue

Value of the request parameter --queue, which determines the queue to which a Spark task is submitted for execution.

default

Advanced attributes

SQL Execution Parameters

When writing data to Hudi, you can configure Spark parameters. The parameters you enter here will be added to the request body as --conf parameters. Spark driver and executor parameters are also supported.

--driver-memory=2G,--executor-memory=2G,--driver-cores=1,--executor-cores=1,--num-executors=2

DB Write Time Field

When a table is automatically created, this field is automatically added to the table creation statement. When the data is written to the Hudi table, the value of this field is the current time. The field must be of the timestamp type.

N/A

Write Parameters

Parameter configured using the set syntax to control the insertion of data into Hudi through a Spark SQL statement

hoodie.combine.before.upsert

Compaction Mode

Compaction execution mode. This parameter takes effect only for MOR tables.

  • Asynchronous compaction: After data is written to Hudi, only a compaction execution plan is generated. Clean and archive operations are not performed. This mode must be used together with asynchronous compaction scripts.
  • Synchronous compaction: The compaction, clean, and archive operations are performed immediately after data is written to the Hudi table, which increases the job execution time. By default, five commits trigger compaction. By default, at least 20 deltacommits are retained, and a maximum of 30 deltacommits are retained. By default, 10 data files corresponding to deltacommits are retained after clean operations.

Asynchronous compaction