Help Center/ MapReduce Service/ Component Development Specifications/ Hudi/ Spark on Hudi Development Specifications/ Suggestions for Configuring Resources for Hudi Data Read and Write
Updated on 2026-02-25 GMT+08:00
View PDF
Share

Suggestions for Configuring Resources for Hudi Data Read and Write

  • Resource configuration rules for Spark to read and write Hudi data are as follows: The ratio of memory to CPU cores is 2:1, and the ratio of off-heap memory to CPU cores is 0.5:1. That is, 2 GB heap memory and 0.5 GB off-heap memory are required for one core.

    During the Spark initialization, the above resource ratios need to be changed to 4:1 and 1:1, respectively, due to a large amount of data being processed.

    Example configuration:

    spark-submit
  --master yarn-cluster
  --executor-cores 2                           --Number of CPU cores of each executor
  --executor-memory 4g                         --Heap memory size of each executor
  --conf spark.executor.memoryOverhead=1024    --Off-heap memory size (MB) of each executor
  • For ETL calculations based on Spark, it is recommended that the ratio of CPU cores to memory be greater than 1:2, preferably ranging from 1:4 to 1:8.

    The previous rules define the resource ratio for read and write operations. If Spark jobs involve both read and write operations and service logic computing, the memory usage increases. In this case, it is recommended that the ratio of CPU cores to memory be greater than 1:2. If the logic is complex, increase the memory size as required. Generally, the recommended ratio of CPU cores to memory is 1:4 to 1:8.

  • For the write resource configuration of the bucket table, it is recommended that the number of CPU cores be at least equal to the number of buckets. Ideally, the suggested number of CPU cores should be calculated as follows: Number of CPU cores = Number of write partitions x Number of buckets. If the actually configured core count is less than this value, the write performance decreases linearly.

    Example configuration:

    The current table has 3 buckets, and there are 2 partitions to be written simultaneously. It is recommended that the number of cores configured for the Spark import task be greater than or equal to 6.

    spark-submit
  --master yarn-cluster
  --executor-cores 2
  --executor-memory 4g
  --executor-num 3

    The provided configuration signifies that executor-num x executor-cores = 6 ≥ Number of partitions x Number of buckets =6.