Submitting a Spark Job as a Proxy User

Scenario

Submit a Spark task as the actual running user or a proxy user. This section describes how to enable the proxy user function to submit Spark tasks.

Prerequisites

The test (actual running user) and test1 (proxy user) users have been created and added to the hadoop (primary group), hive, and supergroup groups.

Submitting a Spark Task as a Proxy User in spark-beeline

1. Modify JDBCServer instance parameters.
   Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations, click All Configurations, click JDBCServer(Role), select Customization, find the spark.core-site.customized.configs parameter, and add the following custom parameters:

   Parameter	Value
   hadoop.proxyuser.test.hosts	*
   hadoop.proxyuser.test.groups	*

   

   

   • In the configuration, test is the actual running user.
   • The value of hadoop.proxyuser.test.hosts is *, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   • The value of hadoop.proxyuser.test.groups is *, which indicates that any proxy user can be used after user test is connected and the proxy user can belong to any user group.

2. Change the values of the following parameters to switch to the JDBCServer multi-instance mode:

   Parameter	Value
   spark.scheduler.allocation.file	#{conf_dir}/fairscheduler.xml
   spark.thriftserver.proxy.enabled	false

3. Save the configuration and restart the Spark service.
4. Log in to the Spark client node and run the following commands:

   cd Client installation directory

   source bigdata_env

   source Spark/component_env

   In security mode, additionally run the following command:

   kinit test Enter the password for authentication. (Change the password upon your first login.)

5. Run the following Beeline commands of Spark to submit a task:

   cd /opt/client/Spark/spark/bin

   ./beeline

   !connect jdbc:hive2://IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client,IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client,IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.hadoop.com@HADOOP.COM;hive.server2.proxy.user=test1

   Specifically:

   • IP address of the node where the ZooKeeper instance is: To obtain the IP address, on FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance.
   • Port number of the ZooKeeper client: To obtain the port number, on FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations then All Configurations, and search for clientPort.
   • hive.server2.proxy.user=test1: test1 is a proxy user.

   

6. Create a Spark table.

   create table sparktest1(a string,b int);

   View the created table.

   desc formatted sparktest1;

   

   You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.

Submitting a Spark Task as a Proxy User in spark-sql and spark-submit

1. Modify the HDFS instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > HDFS. Click Configurations then All Configurations, click HDFS(Service), select Customization, find the hdfs.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value
   hadoop.proxyuser.test.hosts	*
   hadoop.proxyuser.test.groups	*

2. Modify the Yarn instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, click Yarn(Service), select Customization, find the yarn.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value
   hadoop.proxyuser.test.hosts	*
   hadoop.proxyuser.test.groups	*

3. Modify the SparkReource instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations, click SparkResource(Role), select Customization, find the spark.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value
   hadoop.proxyuser.test.hosts	*
   hadoop.proxyuser.test.groups	*

4. Modify the Hive instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Hive. Click Configurations then All Configurations, click Hive(Service), select Customization, find the core.site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value
   hadoop.proxyuser.test.hosts	*
   hadoop.proxyuser.test.groups	*

5. Restart the HDFS, YARN, Spark, and Hive services and update the configuration files on their clients.
6. Log in to the Spark client node and run the following commands:

   cd Client installation directory

   source bigdata_env

   source Spark/component_env

   In security mode, additionally run the following command:

   kinit test Enter the password for authentication. (Change the password upon your first login.)

7. Submit the spark-sql task.

   spark-sql --master yarn --proxy-user test1

8. Create a Spark table.

   create table sparktest2(a string,b int);

   View the created table.

   desc formatted sparktest2;

   

   You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.

9. Use the redelivered client to submit the spark-submit task.

   spark-submit --master yarn --class org.apache.spark.examples.SparkPi --master yarn-client --proxy-user test1 /opt/client/Spark/spark/examples/jars/spark-examples_*.jar

   

10. View information about applications running in YARN.

    

    You can see that the running user of the task is test1, and the proxy user is successfully used.