An Error Is Reported When spark-beeline Is Used to Query a Hive View

Issue

In MRS 3.1.2, an error is reported when spark-beeline is used to query a Hive view. The error information is as follows.

After spark.sql.hive.manageFilesourcePartitions=false is set as prompted, no data can be found. However, data can be queried in Hive.

Cause Analysis

Failed to convert the format. Spark SQL uses its built-in MetaStore instead of Hive MetaStore. As a result, metadata fails to be read. Spark SQL reads data in Parquet format, but Hive reads data in ORC format by default.

Procedure

1. Log in to the Spark client node and run the following commands to access Spark SQL:

   cd Client installation directory

   source bigdata_env

   source Spark2x/component_env

   kinit Component service user (kinit is not required in a normal cluster.)

   spark-sql

2. Run the following command to set spark.sql.hive.convertMetastoreOrc to false:

   set spark.sql.hive.convertMetastoreOrc=false;

3. Query the Hive view again.