Submitting a Spark Job as a Proxy User
Scenarios
Submit a Spark task as the actual running user or a proxy user. This section describes how to enable the proxy user function to submit Spark tasks.
Notes and Constraints
This section applies only to MRS 3.3.0-LTS or later.
Prerequisites
The test (actual running user) and test1 (proxy user) users have been created and added to the hadoop (primary group), hive, and supergroup groups.
Submitting a Spark Task as a Proxy User in spark-beeline
- Modify JDBCServer instance parameters.
Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations and then All Configurations, choose JDBCServer(Role) > Customization, find the spark.core-site.customized.configs parameter, and add the following custom parameters:
Parameter
Value
Description
hadoop.proxyuser.test.hosts
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
hadoop.proxyuser.test.groups
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
- Change the values of the following parameters to switch to the JDBCServer multi-instance mode:
Parameter
Value
Description
spark.scheduler.allocation.file
#{conf_dir}/fairscheduler.xml
Path to the fair scheduler XML file that is shared among multiple tenants and instances.
spark.thriftserver.proxy.enabled
false
Whether to use the multi-tenant mode. false indicates that the multi-instance mode is used. true indicates that the multi-tenant mode is used.
- Save the configuration and restart the Spark service.
- Log in to the Spark client node and run the following commands:
Navigate to the directory where the client is installed.
cd Client installation directory
Load the environment variables.
source bigdata_env
Load the component environment variables.
source Spark/component_env
In security mode, additionally perform the security authentication. Authentication is not required in normal mode.
kinit test
Enter the password for authentication. (Change the password upon your first login.)
- Use Beeline commands of Spark to submit a task.
Go to the Beeline directory of Spark.
cd Client installation directory/Spark/spark/bin
Enable the Beeline command.
./beeline
Submit a task as a proxy user.
!connect jdbc:hive2://IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port,IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port,IP address of the node where the ZooKeeper instance is deployed:ZooKeeper client port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.hadoop.com@HADOOP.COM;hive.server2.proxy.user=test1
Specifically:
- IP address of the node where the ZooKeeper instance is: To obtain the IP address, on FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance.
- Port number of the ZooKeeper client: To obtain the port number, on FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations and then All Configurations, and search for clientPort.
- hive.server2.proxy.user=test1: test1 is a proxy user.
- Create a Spark table.
create table sparktest1(a string,b int);
View the created table.
desc formatted sparktest1;
You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.
Submitting a Spark Task as a Proxy User in spark-sql and spark-submit
- Modify the HDFS instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > HDFS. Click Configurations and then All Configurations, click HDFS(Service), select Customization, find the hdfs.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:
Parameter
Value
Description
hadoop.proxyuser.test.hosts
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
hadoop.proxyuser.test.groups
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
- Modify the Yarn instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, click Yarn(Service), select Customization, find the yarn.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:
Parameter
Value
Description
hadoop.proxyuser.test.hosts
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
hadoop.proxyuser.test.groups
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
- Modify the SparkResource instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations and then All Configurations, click SparkResource(Role), select Customization, find the spark.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:
Parameter
Value
Description
hadoop.proxyuser.test.hosts
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
hadoop.proxyuser.test.groups
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
- Modify the Hive instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Hive. Click Configurations and then All Configurations, click Hive(Service), select Customization, find the core.site.customized.configs parameter, add the following custom parameters, and save the configuration:
Parameter
Value
Description
hadoop.proxyuser.test.hosts
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
hadoop.proxyuser.test.groups
*
In the configuration, test is the actual user, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
- Restart the HDFS, YARN, Spark, and Hive services and update the configuration files on their clients.
- Log in to the Spark client node and run the following commands:
Navigate to the directory where the client is installed.
cd Client installation directory
Load the environment variables.
source bigdata_env
Load the component environment variables.
source Spark/component_env
In security mode, additionally perform the security authentication. Authentication is not required in normal mode.
kinit test
Enter the password for authentication. (Change the password upon your first login.)
- Submit the spark-sql task.
spark-sql --master yarn --proxy-user test1
- Create a Spark table.
create table sparktest2(a string,b int);
View the created table.
desc formatted sparktest2;
You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.
- Use the redelivered client to submit the spark-submit task.
spark-submit --master yarn --class org.apache.spark.examples.SparkPi --master yarn-client --proxy-user test1 Client installation directory/Spark/spark/examples/jars/spark-examples_*.jar
- View information about applications running in YARN.
You can see that the running user of the task is test1, and the proxy user is successfully used.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot