To MRS Hudi

Basic parameters

Destination Link Name

MRS Hudi link

hudi_to_cdm

Database Name

Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed.

dbadmin

Table Name

Click the icon next to the text box. The dialog box for selecting the table is displayed.

This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. You can use macro variables of date and time in a scheduled job to synchronize incremental data periodically. For details, see Using Macro Variables of Date and Time.

cdm

Compaction Mode

Compaction execution mode. This parameter takes effect only for MOR tables.

• Asynchronous compaction: After data is written to Hudi, only a compaction execution plan is generated. Clean and archive operations are not performed. This mode must be used together with asynchronous compaction scripts.
• Synchronous compaction: The compaction, clean, and archive operations are performed immediately after data is written to the Hudi table, which increases the job execution time.

Synchronous Compaction

Write Parameters

Parameter configured using the set syntax to control the insertion of data into Hudi through a Spark SQL statement

hoodie.compact.inline

Auto Table Creation

Whether to automatically create Hudi tables

• Non-auto creation: CDM will not automatically create a table.
• Auto creation: If the destination database does not contain the table specified by Table Name, CDM will automatically create the table. If the table specified by Table Name already exists, no table is created and data is written to the existing table.

Non-auto creation

Location

This parameter is displayed when Auto Table Creation is set to Auto creation.

OBS or HDFS path where database table files are stored

-

Hudi Table Type

This parameter is displayed when Auto Table Creation is set to Auto creation.

Storage type of the Hudi table

• MOR: Data is written to a log file in avro format and then merged into a Parquet file when being read.
• COW: Data is directly written to a Parquet file.

MOR

Hudi table primary key

This parameter is displayed when Auto Table Creation is set to Auto creation.

Primary keys for creating a Hudi table. Use commas (,) to separate multiple keys.

-

Hudi Table Key Generator Class

This parameter is displayed when Auto Table Creation is set to Auto creation.

Primary key generation type, which implements org.apache.hudi.keygen.KeyGenerator to extract key values from input records.

-

Hudi table pre-combine key

This parameter is displayed when Auto Table Creation is set to Auto creation.

If two records have the same primary key, the record with a larger precombine value is retained.

ts

Hudi Table Partition Fields

This parameter is displayed when Auto Table Creation is set to Auto creation.

Partition fields for creating a Hudi table. Use commas (,) to separate multiple fields.

-

Hudi table compression policy (whether to enable write compression)

This parameter is displayed when Auto Table Creation is set to Auto creation.

Policy for compressing data online. This parameter takes effect only for MOR tables.

Yes

Hudi Table Clean Policy (Reserved Submissions)

This parameter is displayed when Auto Table Creation is set to Auto creation.

Number of submissions reserved during clearance

1

Hudi Table Archiving Policy (Minimum Retention Submissions)

This parameter is displayed when Auto Table Creation is set to Auto creation.

Minimum number of submissions retained during archiving

1

Hudi Table Archiving Policy (Maximum Number of Retained Submissions)

This parameter is displayed when Auto Table Creation is set to Auto creation.

Maximum number of submissions retained during archiving

100

Hudi table options

This parameter is displayed when Auto Table Creation is set to Auto creation.

Custom parameters for creating a Hudi table. The parameters will take effect in options.

Primary key, combineKey, index

Clear Data Before Import

Whether the data in the destination table is cleared before data import. The options are as follows:

• Yes: The data is cleared.
• No: The data is not cleared. Instead, it will be added to the existing table.

No

Newline character processing mode

Newline character processing mode. The following options are available:

• Delete
• Replace with another string
• Ignore

Delete

Newline Replacement String

This parameter is available when Newline character processing mode is set to Replace with another string.

It indicates the string used for replacement.

-

Run Queue

This parameter is available only when the connection type is API.

It is the value of the request parameter --queue, which determines the queue to which a Hive task is submitted for execution.

default

SQL Execution Parameters

When compiling Hudi, you can configure Spark parameters. The parameters you enter will be added to the request body as --conf parameters. Spark Driver and Executor parameters are also supported.

--driver-memory=2G,--executor-memory=2G,--driver-cores=1,--executor-cores=1,--num-executors=2

Advanced attributes

Use the import time field

A field marked as the import time field. If a table is automatically created, this field is automatically added to the table creation statement. When data is written to Hudi, the value of this field is replaced by the current time. If the table is not automatically created, select the existing import time field.

Yes

Data import time field name

This parameter is available when Use the import time field is set to Yes.

It specifies the time when data is written to Hudi.

cdc_last_update_date