Configuration Suggestions

Set Traffic Limit When a Hudi Table Is the Source

To prevent job exceptions caused by heavy traffic, set a traffic limit, which should be the peak value of the pressure test for service rollout.

Add the following parameter:

'read.rate.limit' = '1000'

Set execution.checkpointing.tolerable-failed-checkpoints

For Flink On Hudi jobs, set checkpoint tolerance times to a larger value, for example, 100.

Enabling the Hive-Style Partitioning Configuration When Flink Reads and Writes Hudi Partitioned Tables

If the Hive-style partitioning configuration is not enabled, compatibility issues may arise when Spark and Flink work together to process the same Hudi tables.

[Example] Enable the Hive-style partition configuration.

CREATE TABLE stream_mor(
  id int,
  name VARCHAR(20),
  age INT,
  `date` VARCHAR(20)
) PARTITIONED BY (`date`) WITH (
  'connector' = 'hudi',
  'path' = 'hdfs://hacluster/tmp/hudi_mor',
  'table.type' = 'MERGE_ON_READ',
  'hoodie.datasource.write.recordkey.field' = 'id',
  'write.precombine.field' = 'age',
  'index.type' = 'BUCKET',
  'hoodie.datasource.write.hive_style_partitioning' = 'true',
  'hoodie.datasource.hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
  'hoodie.bucket.index.num.buckets' = '4',
  'write.tasks' = '4'
);