Write

No Data Updated

1. By default, the DefaultHoodieRecordPayload is used to calculate the update behavior of Hudi tables. If the preCombine field of incremental data is smaller than that of existing data, the data is not updated. If the preCombine field of incremental data is greater than or equal to that of existing data, the data is updated. Different payloads have different update behaviors.

   

2. The write operation is not committed or fails. As a result, incremental data cannot be read.
   • The write operation failed. The data cannot be read.

     

   • The write operation is successful. The data can be read.

     

3. When different indexes are used to write the same Hudi table, the data is not updated, and the Hudi table is damaged (usually in the scenario where Flink and Spark are used together).

   When multiple indexes are used to write data to a Hudi table, the files are marked with multi-index.write and then an error is reported for the write task.

   

   

4. When multiple ComplexKeyGenerators are used to write data to the same Hudi table, the data is not updated, and the Hudi table is damaged (usually occurred in the secondary development scenario where the attribute verification is bypassed).
   1. View the hoodie.properties file of the Hudi table to check the KeyGenerator class of the Hudi table.

      

   2. Query the value of the _hoodie_record_key metadata field of the Hudi table. If the same data is written using different KeyGenerator classes, the generated values of the _hoodie_record_key are different, which affects the update.

      org.apache.hudi.keygen.ComplexKeyGenerator

      

      org.apache.hudi.keygen.SimpleKeyGenerator

      

Deadlock

Caused by: org.apache.hudi.exception.HoodieLockException: FAILED_TO_ACQUIRE lock at ZkBasePath = /hudi/lock/write, lock key = xxx.db_xxx_table

1. Manually release the ZooKeeper lock in Spark SQL.

   Locate the lock key in the log and run the following SQL statement:

   call remove_lock_on_zookeeper(znode => "/hudi/lock/write/${lock key}");

2. Log in to the MRS client, run the source command to set environment variables, and run the zk-cli.sh command to go to the ZooKeeper background to manually delete the znode.

Duplicate BUCKET ID

Check the error stack:

It is found that the 00000001 bucket in the 2021-12-25 partition has multiple file IDs.

• Scenario 1: Writing the BUCKET index table using insert overwrite

  Each insert overwrite operation generates a replacecommit metadata file, and a Parquet file in the corresponding bucket. The error occurs when the replacecommit file is archived but the Parquet file is not cleaned.

  Solution: Check the settings of the clean and archive parameters and ensure that the clean operation is performed before the archive operation.

  

• Scenario 2: Writing a BUCKET index table using bulk_insert

  For a BUCKET index table, the bulk_insert operation can be performed only once. The error occurs if the operation is performed twice or more. In this case, you need to recreate the table.

• Scenario 3: Task 1 of writing data to Hudi table A and Task 2 of truncating Hudi table A and writing data to Hudi table A

  The error occurs when Task 1 and Task 2 are executed at the same time. In this case, you need to adjust the service dependency to prevent concurrent execution.

Attribute Verification

• The primary key field /preCombine cannot be empty.

  Check whether the values of the primary key fields in the inventory data and incremental data are empty or null.

  

  Check whether the values of the preCombine fields in the inventory data and incremental data are empty or null.

  

• The attribute value in the hoodie.properties file is inconsistent with the value in Table Properties or the value manually specified in the Session.

  The attribute value in Table Properties or Session must be the same as that in the hoodie.properties file. Otherwise, the data cannot be written because these attributes affect data consistency.