On this page

Show all

Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Hudi/ Common Issues About Hudi/ Data Write/ What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?

What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?

Updated on 2022-09-15 GMT+08:00

Question

Hudi consumes much space in a temporary folder during upsert.

Answer

Hudi will spill part of input data to disk if the maximum memory for merge is reached when much input data is upserted.

If the memory is sufficient, increase the memory of the Spark executor and add the hoodie.memory.merge.fraction option, for example, option("hoodie.memory.merge.fraction", "0.8").

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback