Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Spark2x (for MRS 3.x or Later)/ Common Issues About Spark2x/ Spark SQL and DataFrame/ Why Does the --hivevar Option I Specified in the Command for Starting spark-beeline Fail to Take Effect?
Updated on 2023-08-03 GMT+08:00
View PDF
Share

Why Does the --hivevar Option I Specified in the Command for Starting spark-beeline Fail to Take Effect?

Question

The --hivevar option specified in the command for starting spark-beeline fails to take effect.

If --hivevar <VAR_NAME>=<var_value> is used in the command for starting spark-beeline in an MRS cluster to customize a variable, no error is reported when spark-beeline is started. However, when the variable <VAR_NAME> is used in SQL statements, an error message is displayed because <VAR_NAME> cannot be parsed.

The following example describes the scenario:

  1. Run the following command to start the spark-beeline:

    spark-beeline --hivevar <VAR_NAME>=<var_value>

  1. Run the a SQL statement, for example, DROP TABLE ${VAR_NAME} in spark-beeline. The VAR_NAME exception is thrown.

Answer

The multi-session management function is added to the MRS cluster. Hive feature --hivevar <VAR_NAME>=<var_value> is no longer supported in Spark, and the --hivevar option in the spark-beeline startup command is invalid.

Parent topic: Spark SQL and DataFrame