Help Center/ MapReduce Service/ Component Development Specifications/ Hudi/ Spark on Hudi Stream Scenario Operation Guide/ Troubleshooting Methods for Hudi Common Problems/ Performance
Updated on 2026-02-25 GMT+08:00
Performance
How Do I Quickly Determine the Performance Impact Factors of a Hudi Table?
- Check the index.
- Run the desc formatted Database name.Table name command to query the location attribute of the Hudi table, which is the physical path (usually an HDFS path or OBS path) of the Hudi table.
- Check the index information in the hoodie.index.properties file of the .hoodie directory in the physical path of the Hudi table. The index information here is accurate because the index may be incorrectly entered during table creation.
- Check whether the index is proper.
- SIMPLE index is recommended for COW tables in update scenarios (insert into and upsert operations), and BUCKET index is recommended for COW tables in non-update scenarios (insert overwrite operations). BLOOM index is not recommended because it has a higher false positive rate and slower performance as the data is frequently updated.
- BUCKET index is recommended for MOR tables.
- If there are more than 10 million data records, you are advised to use the MOR table + BUCKET index + partitioning. The COW table is not recommended because the SIMPLE index and BLOOM index write a large number of files when there are a large number of data records, and the BUCKET index causes more write amplification issues in the COW table.
- Check the buckets of the BUCKET index.
- Run the desc formatted Database name.Table name command to query the location attribute of the Hudi table, which is the physical path (usually an HDFS path or OBS path) of the Hudi table.
- Check the size of the Parquet file in each bucket in the physical path of the Hudi table. It is recommended that the file size be kept under 300 MB. The larger the file size, the slower the performance. If the file size exceeds 1 GB, the read and write operations are very slow.
- Check the physical directory of the Hudi table.
- Run the desc formatted Database name.Table name command to query the location attribute of the Hudi table, which is the physical path (usually an HDFS path or OBS path) of the Hudi table.
- Check the following items in the Hudi table directory:
- Whether the data is partitioned or not. If there are more than 10 million data records, partition or bucket index must be used to avoid full table update. You can specify partition conditions to update and read data. The more files are updated, the slower the write performance is.
- Number of files and file size. For the SIMPLE index and BLOOM index, check the number of existing files and the update range of incremental data in Hudi tables. For the BUCKET index, check the size of the Parquet files.
- Whether the clean operation on the COW table/MOR table is normal. For details, see Clean. In the insert overwrite scenario, the more historical records are retained, the slower the read and write performance is.
- Whether the archive operation on the COW/MOR table is normal. For details, see Archive. The more metadata in the .hoodie directory, the slower the read and write performance.
- Whether the compaction of the MOR table is normal. The more log files, the slower the read and write performance.
Parent topic: Troubleshooting Methods for Hudi Common Problems
What is your overall rating for this page?
0
1
2
3
4
5
6
7
8
9
10
Very dissatisfiedVery satisfied
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
The system is busy. Please try again later.
For any further questions, feel free to contact us through the chatbot.
Chatbot