Task Failed Due to Concurrent Data Overwrites on the Same Table or Partition

Symptom

When Hive executes an INSERT statement, an error is reported indicating that a file or directory already exists or is cleared in HDFS. The error details are as follows:

Cause Analysis

1. Check the start time and end time of the task based on the HiveServer audit logs.
2. Check whether data is inserted into the same table or partition in the time segment.
3. Concurrent data overwrites to the same table or partition in Hive may cause tasks to fail because one task moves the data of another task when multiple tasks operate the same temporary data directory.

Solution

The service logic is modified so that data is inserted to the same table or partition in single thread mode.