Updated on 2026-11-12 GMT+08:00

From Hudi

MRS Spark JDBC Server cannot intercept submitted SQL statements. If a Hudi connection uses the JDBC mode, SQL statements submitted after the job is stopped may fail to be canceled.

Table 1 Parameter description

Type

Parameter

Description

Example Value

Basic parameters

Database Name

Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed.

default

Table Name

Hudi table name. Click the icon next to the text box. The dialog box for selecting the table is displayed.

This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. For details, see Incremental Synchronization Using the Macro Variables of Date and Time.

NOTE:

If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).

TBL_E

Run Queue

Value of the request parameter --queue, which determines the queue to which a Spark task is submitted for execution.

default

Advanced attributes

SQL Execution Parameters

When writing data to Hudi, you can configure Spark parameters. The parameters you enter here will be added to the request body as --conf parameters. Spark driver and executor parameters are also supported.

--driver-memory=2G,--executor-memory=2G,--driver-cores=1,--executor-cores=1,--num-executors=2

Preceding Parameters

Parameters configured using the SET syntax to control the extraction of data from a Hudi table through a Spark SQL statement. The following parameters are supported:

  • hoodie.metadata.enable=true: enables the Hudi metadata table to avoid full file scanning.
  • spark.sql.shuffle.partitions=200: controls the number of shuffle partitions. You can increase the value for large tables to improve the parallelism.
  • spark.sql.files.maxPartitionByte=134217728: maximum number of bytes to be packed into a single partition during file reading (128 MB). You can increase the value to reduce the number of partitions.

-

Where Clause

This parameter indicates the where clause to be extracted. If this parameter is not set, the entire table is extracted. If the table to be migrated does not contain the fields specified by the where clause, the migration will fail.

You can set a date macro variable to extract data generated on a specific date. For details, see Incremental Migration of Relational Databases.

NOTE:

If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).

age > 18 and age <= 60