Help Center/ MapReduce Service/ Component Operation Guide (ME-Abu Dhabi Region)/ Using Hudi/ Common Issues About Hudi/ Data Write/ What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?
Updated on 2024-07-19 GMT+08:00

What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?

Question

Hudi consumes much space in a temporary folder during upsert.

Answer

Hudi will spill part of input data to disk if the maximum memory for merge is reached when much input data is upserted.

If the memory is sufficient, increase the memory of the Spark executor and add the hoodie.memory.merge.fraction option, for example, option("hoodie.memory.merge.fraction", "0.8").