Configuring Spark Tasks Not to Obtain HBase Token Information
Scenarios
When Spark is used to submit tasks, the driver obtains tokens from HBase by default. To access HBase, you need to configure the jaas.conf file for security authentication. If the jaas.conf file is not configured, the application will fail to run.
Therefore, perform the following operations based on whether the application involves HBase:
- If the application does not involve HBase, you do not need to obtain the HBase tokens. In this case, set spark.yarn.security.credentials.hbase.enabled to false.
- If the application involves HBase, set spark.yarn.security.credentials.hbase.enabled to true and configure the jaas.conf file on the driver as follows:
{client}/spark/bin/spark-sql --master yarn-client --principal {principal} --keytab {keytab} --driver-java-options "-Djava.security.auth.login.config={LocalPath}/jaas.conf"
Specify the Keytab and Principal in the jaas.conf file. The following is an example:
Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab = "{LocalPath}/user.keytab" principal="super@<System domain name>" useTicketCache=false debug=false; };
Configuration
- Install the Spark client.
For details, see Installing a Client.
- Log in to the Spark client node as the client installation user.
Modify the following parameters in the {Client installation directory}/Spark/spark/conf/spark-defaults.conf file on the Spark client.
Table 1 Parameter description Parameter
Description
Example Value
spark.yarn.security.credentials.hbase.enabled
Indicates whether HBase obtains a token.
- true: HBase obtains a token.
- false: HBase does not obtain a token.
false
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot