Submitting a Spark Job as a Proxy User

Scenarios

Submit a Spark task as the actual running user or a proxy user. This section describes how to enable the proxy user function to submit Spark tasks.

Notes and Constraints

This section applies only to MRS 3.3.0-LTS or later.

Prerequisites

The test (actual running user) and test1 (proxy user) users have been created and added to the hadoop (primary group), hive, and supergroup groups.

Submitting a Spark Task as a Proxy User in spark-beeline

1. Modify JDBCServer instance parameters.
   Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations and then All Configurations, choose JDBCServer(Role) > Customization, find the spark.core-site.customized.configs parameter, and add the following custom parameters:

   Parameter	Value	Description
   hadoop.proxyuser.test.hosts	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   hadoop.proxyuser.test.groups	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.

   

2. Change the values of the following parameters to switch to the JDBCServer multi-instance mode:

   Parameter	Value	Description
   spark.scheduler.allocation.file	#{conf_dir}/fairscheduler.xml	Path to the fair scheduler XML file that is shared among multiple tenants and instances.
   spark.thriftserver.proxy.enabled	false	Whether to use the multi-tenant mode. false indicates that the multi-instance mode is used. true indicates that the multi-tenant mode is used.

3. Save the configuration and restart the Spark service.
4. Log in to the Spark client node and run the following commands:
   Navigate to the directory where the client is installed.

   cd Client installation directory

   Load the environment variables.

   source bigdata_env

   Load the component environment variables.

   source Spark/component_env

   In security mode, additionally perform the security authentication. Authentication is not required in normal mode.

   kinit test

   Enter the password for authentication. (Change the password upon your first login.)

5. Use Beeline commands of Spark to submit a task.
   Go to the Beeline directory of Spark.

   cd Client installation directory/Spark/spark/bin

   Enable the Beeline command.

   ./beeline

   Submit a task as a proxy user.

   !connect jdbc:hive2://IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port,IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port,IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.hadoop.com@HADOOP.COM;hive.server2.proxy.user=test1

   Specifically:

   • IP address of the node where the ZooKeeper instance is: To obtain the IP address, on FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance.
   • Port number of the ZooKeeper client: To obtain the port number, on FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations and then All Configurations, and search for clientPort.
   • hive.server2.proxy.user=test1: test1 is a proxy user.

     

6. Create a Spark table.

   create table sparktest1(a string,b int);

   View the created table.

   desc formatted sparktest1;

   

   You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.

Submitting a Spark Task as a Proxy User in spark-sql and spark-submit

1. Modify the HDFS instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > HDFS. Click Configurations and then All Configurations, click HDFS(Service), select Customization, find the hdfs.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value	Description
   hadoop.proxyuser.test.hosts	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   hadoop.proxyuser.test.groups	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.

2. Modify the Yarn instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, click Yarn(Service), select Customization, find the yarn.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value	Description
   hadoop.proxyuser.test.hosts	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   hadoop.proxyuser.test.groups	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.

3. Modify the SparkResource instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations and then All Configurations, click SparkResource(Role), select Customization, find the spark.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value	Description
   hadoop.proxyuser.test.hosts	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   hadoop.proxyuser.test.groups	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.

4. Modify the Hive instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Hive. Click Configurations and then All Configurations, click Hive(Service), select Customization, find the core.site.customized.configs parameter, add the following custom parameters, and save the configuration:

   Parameter	Value	Description
   hadoop.proxyuser.test.hosts	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
   hadoop.proxyuser.test.groups	*	In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.

5. Restart the HDFS, YARN, Spark, and Hive services and update the configuration files on their clients.
6. Log in to the Spark client node and run the following commands:
   Navigate to the directory where the client is installed.

   cd Client installation directory

   Load the environment variables.

   source bigdata_env

   Load the component environment variables.

   source Spark/component_env

   In security mode, additionally perform the security authentication. Authentication is not required in normal mode.

   kinit test

   Enter the password for authentication. (Change the password upon your first login.)

7. Submit the spark-sql task.

   spark-sql --master yarn --proxy-user test1

8. Create a Spark table.

   create table sparktest2(a string,b int);

   View the created table.

   desc formatted sparktest2;

   

   You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.

9. Use the redelivered client to submit the spark-submit task.

   spark-submit --master yarn --class org.apache.spark.examples.SparkPi --master yarn-client --proxy-user test1 Client installation directory/Spark/spark/examples/jars/spark-examples_*.jar

   

10. View information about applications running in YARN.

    

    You can see that the running user of the task is test1, and the proxy user is successfully used.