To MRS Hudi
If the destination link of a job is an MRS Hudi link, configure the destination job parameters based on Table 1.
General Configuration |
||
---|---|---|
Item |
Configuration Description |
Recommended Configuration |
Destination Link Name |
MRS Hudi link |
hudi_to_cdm |
Database Name |
Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed. |
dbadmin |
Table Name |
Click the icon next to the text box. The dialog box for selecting the table is displayed. This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. You can use macro variables of date and time in a scheduled job to synchronize incremental data periodically. For details, see Using Macro Variables of Date and Time.
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
cdm |
Auto Table Creation |
Whether to automatically create Hudi tables
|
Non-auto creation |
Clear Data Before Import |
Whether the data in the destination table is cleared before data import. The options are as follows:
|
No |
Full Data Mode to Write Hoodie |
Hoodie write mode. The default value is Yes, indicating the full mode. Value No indicates the microbatch mode.
NOTE:
This mode cannot be changed during a retry upon failure. |
Yes |
Batch Size |
This parameter is available when Full Data Mode to Write Hoodie is set to No. It specifies the number of data rows written to Hoodie in a single batch. The default value is 100000. |
100000 |
Use the import time field |
A field marked as the import time field. If a table is automatically created, this field is automatically added to the table creation statement. When data is written to Hudi, the value of this field is replaced by the current time. If the table is not automatically created, select the existing import time field. |
Yes |
Data import time field name |
This parameter is available when Use the import time field is set to Yes. It specifies the time when data is written to Hudi.
NOTE:
|
cdc_last_update_date |
Hudi Table Creation Configuration |
||
Location |
OBS or HDFS path where database table files are stored |
- |
Hudi Table Type |
Storage type of the Hudi table
|
MOR |
Hudi table primary key |
Primary keys for creating a Hudi table. Use commas (,) to separate multiple keys. |
- |
Hudi Table Key Generator Class |
Primary key generation type, which implements org.apache.hudi.keygen.KeyGenerator to extract key values from input records. |
- |
Hudi table pre-combine key |
If two records have the same primary key, the record with a larger precombine value is retained.
NOTE:
If no time field is available, you can set a field that is the same as the primary key. When a primary key conflict occurs, the latest record is retained. |
ts |
Hudi Table Partition Fields |
Partition fields for creating a Hudi table. Use commas (,) to separate multiple fields. |
- |
Hudi table compression policy (whether to enable write compression) |
Policy for compressing data online. This parameter takes effect only for MOR tables. |
Yes |
Hudi Table Clean Policy (Reserved Submissions) |
Number of submissions reserved during clearance |
1 |
Hudi Table Archiving Policy (Minimum Retention Submissions) |
Minimum number of submissions retained during archiving |
1 |
Hudi Table Archiving Policy (Maximum Number of Retained Submissions) |
Maximum number of submissions retained during archiving |
100 |
Hudi table options |
Custom parameters for creating a Hudi table. The parameters take effect in options, for example, primary key, combineKey, or index. |
- |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot