Why Do spark-sql and spark-submit Fail to Execute When Ranger Authentication Is Used and the Client Is Mounted in Read-Only Mode?

Question

When Ranger authentication is used and the client is mounted in read-only mode, spark-sql and spark-submit fail to execute, and an error message is displayed, indicating that saving roles to the sparkSql_Hive_roles.json file fails.

Possible Causes

When submitting an application, the Spark client reads the latest Ranger authentication policy file, caches it locally, and updates the $SPARK_HOME/conf/sparkSql_Hive.json and $SPARK_HOME/conf/sparkSql_Hive_roles.json files. In read-only mode, the client configuration files cannot be updated. As a result, an error is reported.

Solution

Method 1: Change the Ranger authentication mode to ACL authentication. For details, see SparkSQL Permission Management(Security Mode).

Method 2: Change the path for storing the policy file and add the modification permission.

On the client, change the ranger.plugin.spark.policy.cache.dir value in the /opt/client/Spark/spark/conf/ranger-spark-security.xml file to a directory that is not on the client, and the directory has the execution permission on Spark.