Help Center/ MapReduce Service/ Component Development Specifications/ Hudi/ Spark on Hudi Stream Scenario Operation Guide/ Troubleshooting Methods for Hudi Common Problems/ Compaction
Updated on 2026-02-25 GMT+08:00
View PDF
Share

Compaction

How Do I Determine Whether a Compaction Is Being Executed?

Run the show compaction on <database name>.<table name> command.

  • For versions earlier than MRS 3.5.0-LTS:

    Analysis results:

    1. If the returned result is empty, compaction is not configured, and compaction has not been executed on the Hudi table.
    2. If the value in the second column of the returned result is compaction, the first compaction plan is not executed.
    3. If the value in the second column of the returned result is commit, the compaction plan has been executed.
  • For MRS 3.5.0-LTS and later versions:

    Analysis results:

    1. If the returned result is empty, compaction is not configured, and compaction has not been executed on the Hudi table.
    2. If the value in the second column of the returned result is requested, the first compaction plan is not executed.
    3. If the value in the second column of the returned result is completed, the compaction plan has been executed.

How Do I Determine Whether Compaction Is Properly Set?

Criteria:

Randomly find a recently written Parquet file. In the Parquet file name, the content before the first underscore (_) is the file ID, and the content after the second underscore (_) is the commit time of the Parquet file. The same rule applies to log files. Find the Parquet file and log file with the same file ID. Then find the Parquet file with the largest commit time, record the commit time as A, and find the log files whose commit time is greater than or equal to A.

  1. If there is only one log file and the size is less than 300 MB, compaction is proper and in the optimal state.
  2. If there are multiple log files and the size of each log file is 1 GB, the compaction is improper and the read performance is affected.

Example

Hudi table:

Analysis result: 00000000-e42f-4a07-be73-23c50a635ed3-0 corresponds to two parquet files. The largest timestamp is 20250328141314559. No log file whose timestamp is greater than or equal to this timestamp is found. 00000001-919b-4bc2-88b7-6dff8408f3eb-0 corresponds to a parquet file. There is only one log file whose timestamp is greater than or equal to 20250328141507479, but the size is 931 bytes. Therefore, compaction is proper.