Clustering

Introduction to Clustering

Clustering reorganizes data layout to improve query performance without affecting the ingestion speed.

Hudi provides different operations, such as insert, upsert, and bulk_insert, through its write client API to write data to a Hudi table. To weight between file size and speed of importing data into the data lake, Hudi provides hoodie.parquet.small.file.limit to configure the minimum file size. You can set it to 0 to force new data to be written to new file groups, or to a higher value to ensure that new data is "padded" to existing small file groups until it reaches the specified size, but this increases ingestion latency.

To support fast ingestion without affecting query performance, the clustering service is introduced to rewrite data to optimize the layout of Hudi data lake files.

The clustering service can run asynchronously or synchronously. It adds a new operation type called REPLACE, which will mark the clustering operation in the Hudi metadata timeline.

Clustering service is based on the MVCC design of Hudi to allow new data to be inserted. Clustering operations run in the background to reformat data layout, ensuring snapshot isolation between concurrent readers and writers.

Clustering is divided into two parts:

• Scheduling clustering: Create a clustering plan using a pluggable clustering strategy.
  1. Identify files that are eligible for clustering: Depending on the selected clustering strategy, the scheduling logic will identify the files eligible for clustering.
  2. Group files that are eligible for clustering based on specific criteria. The data size of each group must be a multiple of targetFileSize. Grouping is a part of the strategy defined in the plan. Additionally, there is an option to control group size to improve parallelism and avoid shuffling large volumes of data.
  3. Save the clustering plan to the timeline in Avro metadata format.
• Execute clustering: Process the plan using an execution strategy to create new files and replace old files.
  1. Read the clustering plan and get clusteringGroups that marks the file groups to be clustered.
  2. Instantiate appropriate strategy class for each group using strategyParams (for example, sortColumns) and apply the strategy to rewrite data.
  3. Create a REPLACE commit and update the metadata in HoodieReplaceCommitMetadata.

Using Clustering

1. Executing clustering synchronously

   Add the following configuration parameters when the data write operation is performed:

   option("hoodie.clustering.inline", "true").

   option("hoodie.clustering.inline.max.commits", "4").

   option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824").

   option("hoodie.clustering.plan.strategy.small.file.limit", "629145600").

   option("hoodie.clustering.plan.strategy.sort.columns", "column1,column2").

2. Executing clustering asynchronously

   Run the following spark-sql command to perform clustering. For details, see CLUSTERING.

   To execute clustering asynchronously, run the set command to set the following parameters:

   set hoodie.clustering.async.enabled = true;

   set hoodie.clustering.async.max.commits = 4;

3. Specifying the ordering mode and sequence of clustering

   Currently, clustering supports three sorting modes: Linear, Z-Order, and Hilbert, which can be configured in option or set mode.

   • Linear ordering: a common but default ordering mode, which applies to ordering one field or multiple low-level fields.
   • Z-order or Hilbert: multi-dimensional ordering, which is available when you set hoodie.layout.optimize.strategy to z-order or hilbert.

     These two ordering modes apply to sorting 2 to 4 fields, for example, multiple fields involved in a query condition.

     Hilbert has a better multi-dimensional ordering effect than Z-order but lower ordering efficiency.

For details, see Common Hudi Parameters.

1. The sorting column of clustering cannot be null. This is restricted by Spark RDD.
2. If the value of target.file.max.bytes is large, increase the value of --spark-memory to execute clustering. Otherwise, the executor memory overflow occurs.
3. Currently, the cleaning operation cannot be performed to delete junk files generated after the clustering fails.
4. After the clustering, sizes of new files may be different, causing data skew.
5. Clustering and upsert operations cannot be performed at the same time.
6. If the clustering is in the inflight state, the files in the file group do not support the update operation.
7. If there is an unfinished clustering plan, an error will be reported when the compaction scheduling plan is generated upon subsequent writing. You must execute the clustering plan in a timely manner.