How Do I Configure a Spark Periodic Task for Hudi Compaction?

Symptom

When writing data to Hudi, DataArts Migration splits compaction tasks into Spark jobs and sends the Spark jobs to MRS for execution.

Solution

1. Modify the configuration of the real-time migration job.

   Disable asynchronous compaction, deletion of historical data files, and archiving for the migration job by configuring the parameters in Global Configuration of Hudi Table Attributes or Edit Table Attribute.

   Table 1 Hudi table parameters

   Parameter	Value	Description
   compaction.schedule.enabled	true	Compaction plan generation is enabled.
   compaction.delta_commits	60	Period for triggering compaction generated by the compaction plan
   compaction.async.enabled	false	Asynchronous compaction is disabled.
   clean.async.enabled	false	Data files of historical versions will be deleted.
   hoodie.archive.automatic	false	Aging of Hudi commit files is enabled.

   Figure 1 Disabling migration compaction
   

   After the preceding parameters are set, no compaction task is executed after the job is started. Compaction plans are generated periodically. You can run the run compaction on command to execute compaction plans.

   

   Compaction plans must be generated by migration tasks and then executed by Spark. Otherwise, a Hudi timeline conflict will occur, causing the Spark compaction job to fail.

2. Create a periodic compaction task for Spark SQL.
   1. Go to the DataArts Factory console and create a Spark SQL job by following the instructions in Developing a Batch Processing Single-Task SQL Job.
      Figure 2 Creating a single-task Spark SQL job
      
   2. Select the Spark data connection corresponding to Hudi and select the database to which the Hudi table belongs.
      Figure 3 Configuring the connection and database
      
   3. Configure the compaction scheduling period.
      Figure 4 Configuring the scheduling period
      
   4. Enter the compaction statements of Spark SQL, and submit and run the job.

      set hoodie.compact.inline = true;
      set hoodie.run.compact.only.inline = true;
      set hoodie.clean.automatic = false;
      set hoodie.cleaner.commits.retained = 120;
      set hoodie.keep.min.commits = 121;
      set hoodie.keep.max.commits = 141; 
      run compaction on `db_name`.`table_name`; 
      run clean on `db_name`.`table_name`;
      run archivelog on `db_name`.`table_name`;

      Figure 5 Submitting and running the job