Error Reported When the WHERE Condition Is Used to Query Tables with Excessive Partitions in FusionInsight Hive

Question

When a table with more than 32,000 partitions is created in Hive, an exception occurs during the query with the WHERE partition. In addition, the exception information printed in metastore.log contains the following information:

Caused by: java.io.IOException: Tried to send an out-of-range integer as a 2-byte value: 32970
        at org.postgresql.core.PGStream.SendInteger2(PGStream.java:199)
        at org.postgresql.core.v3.QueryExecutorImpl.sendParse(QueryExecutorImpl.java:1330)
        at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1601)
        at org.postgresql.core.v3.QueryExecutorImpl.sendParse(QueryExecutorImpl.java:1191)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:346)

Answer

During a query with partition conditions, HiveServer optimizes the partitions to avoid full table scanning. All partitions whose metadata meets the conditions need to be queried. However, the sendOneQuery interface provided by GaussDB limits the parameter value to 32767 in the sendParse method. If the number of partition conditions exceeds 32767, an exception occurs. If you have to query a large number of partitions in a single SQL statement, see Procedure.

Procedure

Log in to FusionInsight Manager, choose Clusters > Services > Hive. On the displayed page, click the Configurations tab and select All Configurations. Choose MetaStore(Role) > Customization, and add metastore.direct.sql.batch.size and its value 10000 to the parameter file hivemetastore-site.xml.
Click Save. In the dialog box that is displayed, click OK.
Click the Instance tab, select all MetaStore instances, click More > Restart Instance, enter the administrator password, and click OK to restart the MetaStore instances.