What Should I Do If the String of a Single Field Is Too Long During Writing of a Job to a Destination Hive ORC Table and the Write Operation Is Blocked or Spark Fails to Read the Data After the Write Operation?

Symptom

During writing of a job to a Hive ORC table, if the string of a single field is too long, the write operation may be blocked or Spark may fail to read the data after the write operation.

Figure 1 Spark fails to read data

Possible Causes

• Scenario 1:

  If the ORC source code uses string dictionary encoding to write strings, the field encoding uses a byte array to cache data. If the total byte size of a batch of data (1,000 pieces by default) exceeds 2 GB, the byte array cache overflows, triggering an out of memory (OOM) exception and resulting in an infinite loop in the ORC source code.

• Scenario 2:

  If the ORC source code uses direct string encoding to write strings, string data is directly written to files. However, if the byte size of the same field in a batch of data (1,000 pieces by default) exceeds 2 GB, the data is written to the same ORC stripe. The byte array from which Spark reads data overflows, triggering a read exception.

Solution

Add the following parameters to the attributes of the Hive connection:

• hive.batch.size=10

  This parameter specifies the size of a batch of data written by the ORC writer. You can reduce the batch size to avoid the issue in scenario 1.

• orc.rows.between.memory.checks=1

  This parameter specifies the number of data records written before ORC checks the data size. Each time ORC writes a batch of data, if the total number of rows in the batch exceeds this value, the total data size will be checked to determine the size of the stripe to which data is written at a time. If the data size exceeds the stripe size (64 MB by default), a new stripe will be generated. This configuration can be used to avoid the issue in scenario 2. If this parameter is set to 1, the stripe size will be checked for each batch.