Optimistic Lock Concurrent Commit Conflicts

Cause of optimistic lock concurrent commit conflicts: Multiple modification jobs (including Insert, Update, Delete, MergeInto, and table service operations) are simultaneously submitted on LakeFormation, leading to concurrent commit conflicts. For example:

In the preceding figure, transaction 1 and transaction 2 are two DML transactions. Transaction 1 processes and commits data based on metadata version 1 and updates the metadata version to version 2. However, Transaction 2 had already started data processing based on metadata version 1 before transaction 1 was committed. During its initial commit attempt, LakeFormation rejects the commit because the metadata has already been updated to version 2. Transaction 2 must then update the metadata and resubmit it.

In such scenarios, if the concurrency volume is extremely high or the conflict detection time for a particular transaction is excessively long, it can lead to an endless cycle of "commit conflict -> update metadata -> detect conflict -> resubmit", resulting in business conflict errors.

To address these issues, Iceberg introduces four parameters.