Selecting a Proper Table Service Execution Mode

Method 1: Executing the Table Service and Write Task Together

Execute the table service and write task together according to the template. Do not delete, add, or modify the parameters in the template.

1. insert overwrite scenario.
   • In this scenario, only clean and archive are required for COW and MOR tables. Compaction is not required.
   • Use COW tables.
     Use the following template:

     set hoodie.cleaner.commits.retained = 5; // clean is enabled by default. In the insert overwrite scenario, the number of versions to be retained cannot be too large. The value 5 is sufficient.
     set hoodie.keep.min.commits = 6; // archive is enabled by default. The minimum number of metadata records to be retained is greater than the number of versions retained by clean.
     set hoodie.keep.max.commits = 7; // Maximum number of metadata records that can be retained by archive.
     insert overwrite SinkTable select * from SourceTable;

2. insert into
   • For COW tables:
     In the insert into scenario, no parameter needs to be set.

     insert into SinkTable select * from SourceTable;

   • For MOR tables:
     In the insert into scenario, only the compaction triggering frequency needs to be considered.

     set hoodie.compact.inline.max.delta.commits = X; // The default value is 5. The service side needs to consider the size of the log file that is appended to the Hudi table after data is written to the table for X times. The optimal performance is to ensure that the size of the log file is less than 300 MB. For example, if the size of the log file is 500 MB after data is written to the table for 10 times, and you need to keep the log file size to 100 MB, you can set the value of this parameter to 100/ (500/10) = 2.
     set hoodie.cleaner.commits.retained = X + 1; 
     set hoodie.keep.min.commits = X + 10;
     set hoodie.keep.max.commits = X + 20;
     insert into SinkTable select * from SourceTable;

Method 2: Executing the Table Service and Write Task Separately

Strictly follow the template. Do not delete, add, or modify parameters in the template.

1. Configured in the write task.
   • For COW tables:

     Perform operations in Method 1: Executing the Table Service and Write Task Together.

   • For MOR tables:

     set hoodie.compact.inline = true;
     set hoodie.schedule.compact.only.inline=true;
     set hoodie.compact.inline.max.delta.commits = X; // The default value is 5. The service side needs to consider the size of the log file that is appended to the Hudi table after data is written to the table for X times. The optimal performance is to ensure that the size of the log file is less than 300 MB. For example, X is 10, when the size of the log file is 500 MB after data is written to the table for 10 times, and you need to keep the log file size to 100 MB, you can set the value of this parameter to 100/ (500/10) = 2.
     set hoodie.clean.automatic = false;
     set hoodie.archive.automatic = false;
     insert into SinkTable select * from SourceTable;

2. Execute the table service using an asynchronous script.

   Generally, the asynchronous script needs to be scheduled on the DataArts Studio platform. If high read performance is required, it is recommended that the script be scheduled once an hour. In other cases, the script can be scheduled once every 2 to 4 hours.

   For MOR tables:

   set hoodie.clean.automatic = false;
   set hoodie.clean.async = false;
   set hoodie.archive.automatic = false;
   set hoodie.archive.async = false; 
   set hoodie.compact.inline = true;
   set hoodie.run.compact.only.inline = true;
   set hoodie.cleaner.commits.retained = X + 1;
   set hoodie.keep.min.commits = X + 10;
   set hoodie.keep.max.commits = X + 20;
   run compaction on <database name>. <table name>;   -- Execute the compaction plan.
   run clean on <database name>. <table name>;        -- Execute the clean operation.
   run archivelog on <database name>.<table name>;    -- Execute the archive operation.

Table Service Selection Skills

• Select Method 1: Executing the Table Service and Write Task Together for COW tables.
• Select Method 1: Executing the Table Service and Write Task Together for insert overwrite operation, regardless of whether the table is a COW or MOR table.
• Data is written to MOR tables using the insert into operation. If the service side can accept that the task duration increases by at least one to two times because compaction, clean, or archive operation is triggered after data is written to a MOR table for X times, select Method 1: Executing the Table Service and Write Task Together. Otherwise, select Method 2: Executing the Table Service and Write Task Separately.
• In the Spark/Flink streaming scenario, select Method 2: Executing the Table Service and Write Task Separately.
• In the scenario where Flink CDC writes data to Hudi MOR tables, select Method 2: Executing the Table Service and Write Task Separately. In addition, changelog is enabled in the CDC scenario. Therefore, you need to add set hoodie.allow.operation.metadata.field = true to the asynchronous table service script.