On this page

Scenario
Prerequisites
Submitting a Spark Task as a Proxy User in spark-beeline
Submitting a Spark Task as a Proxy User in spark-sql and spark-submit

Show all

Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Spark/Spark2x/ Submitting a Spark Job as a Proxy User

Submitting a Spark Job as a Proxy User

Updated on 2024-10-09 GMT+08:00

View PDF

NOTE:

This section applies only to MRS 3.3.0 and later versions.

Scenario

Submit a Spark task as the actual running user or a proxy user. This section describes how to enable the proxy user function to submit Spark tasks.

Prerequisites

The test (actual running user) and test1 (proxy user) users have been created and added to the hadoop (primary group), hive, and supergroup groups.

Submitting a Spark Task as a Proxy User in spark-beeline

Modify JDBCServer instance parameters.

Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations, click All Configurations, click JDBCServer(Role), select Customization, find the spark.core-site.customized.configs parameter, and add the following custom parameters:

Parameter	Value
hadoop.proxyuser.test.hosts	*
hadoop.proxyuser.test.groups	*

Click to enlarge

NOTE:

In the configuration, test is the actual running user.
The value of hadoop.proxyuser.test.hosts is *, which indicates that any proxy user can be used after user test is connected and the number of cluster nodes is not limited.
The value of hadoop.proxyuser.test.groups is *, which indicates that any proxy user can be used after user test is connected and the proxy user can belong to any user group.

Change the values of the following parameters to switch to the JDBCServer multi-instance mode:

Parameter	Value
spark.scheduler.allocation.file	#{conf_dir}/fairscheduler.xml
spark.thriftserver.proxy.enabled	false

Save the configuration and restart the Spark service.
Log in to the Spark client node and run the following commands:

cd Client installation directory

source bigdata_env

source Spark/component_env

In security mode, additionally run the following command:

kinit test Enter the password for authentication. (Change the password upon your first login.)
Run the following Beeline commands of Spark to submit a task:

cd /opt/client/Spark/spark/bin

./beeline

!connect jdbc:hive2://IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client,IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client,IP address of the node where the ZooKeeper instance is:Port number of the ZooKeeper client/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.hadoop.com@HADOOP.COM;hive.server2.proxy.user=test1

Specifically:
- IP address of the node where the ZooKeeper instance is: To obtain the IP address, on FusionInsight Manager, choose Cluster > Services > ZooKeeper and click Instance.
- Port number of the ZooKeeper client: To obtain the port number, on FusionInsight Manager, choose Cluster > Services > ZooKeeper, click Configurations then All Configurations, and search for clientPort.
- hive.server2.proxy.user=test1: test1 is a proxy user.
Create a Spark table.

create table sparktest1(a string,b int);

View the created table.

desc formatted sparktest1;

You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.

Submitting a Spark Task as a Proxy User in spark-sql and spark-submit

Modify the HDFS instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > HDFS. Click Configurations then All Configurations, click HDFS(Service), select Customization, find the hdfs.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

Parameter	Value
hadoop.proxyuser.test.hosts	*
hadoop.proxyuser.test.groups	*

Modify the Yarn instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, click Yarn(Service), select Customization, find the yarn.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

Parameter	Value
hadoop.proxyuser.test.hosts	*
hadoop.proxyuser.test.groups	*

Modify the SparkReource instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations, click SparkResource(Role), select Customization, find the spark.core-site.customized.configs parameter, add the following custom parameters, and save the configuration:

Parameter	Value
hadoop.proxyuser.test.hosts	*
hadoop.proxyuser.test.groups	*

Modify the Hive instance configuration. Log in to FusionInsight Manager and choose Cluster > Services > Hive. Click Configurations then All Configurations, click Hive(Service), select Customization, find the core.site.customized.configs parameter, add the following custom parameters, and save the configuration:

Parameter	Value
hadoop.proxyuser.test.hosts	*
hadoop.proxyuser.test.groups	*

Restart the HDFS, YARN, Spark, and Hive services and update the configuration files on their clients.
Log in to the Spark client node and run the following commands:

cd Client installation directory

source bigdata_env

source Spark/component_env

In security mode, additionally run the following command:

kinit test Enter the password for authentication. (Change the password upon your first login.)
Submit the spark-sql task.

spark-sql --master yarn --proxy-user test1
Create a Spark table.

create table sparktest2(a string,b int);

View the created table.

desc formatted sparktest2;

You can see that Owner of the table is the proxy user test1, and the proxy user is successfully used.
Use the redelivered client to submit the spark-submit task.

spark-submit --master yarn --class org.apache.spark.examples.SparkPi --master yarn-client --proxy-user test1 /opt/client/Spark/spark/examples/jars/spark-examples_*.jar
View information about applications running in YARN.

You can see that the running user of the task is test1, and the proxy user is successfully used.

Parent topic: Using Spark/Spark2x

Previous topic: Accessing the Spark Web UI

Next topic: Configuring Spark to Read HBase Data

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot