Updated on 2022-09-26 GMT+08:00

Configuring the MRS Client

Go to /opt/client/Spark2x/spark/conf (Spark client installation directory) and configure the following parameters in the spark-defaults.conf file:

spark.pyspark.driver.python=/usr/bin/python3
spark.yarn.dist.archives=hdfs://hacluster/user/python/python36.zip#Python