Configuring Security Authentication for Spark Applications

Updated on 2024-08-10 GMT+08:00

View PDF

Overview

In a cluster with the security mode enabled, the components must be mutually authenticated before communicating with each other to ensure communication security.

When developing Spark applications, you may need Spark to communicate with Hadoop and HBase. Code for security authentication needs to be written into the Spark applications to ensure that the applications can work properly.

Three security authentication modes are available:

Authentication by running command lines
To authenticate, run this command in the Spark client before submitting the application for running or logging in to Spark SQL using the CLI:

kinitComponent service user
Authentication by configuring parameters:
You can use any of the following methods to specify the security authentication information.
- Configure the spark.kerberos.keytab and spark.kerberos.principal parameters in the spark-defaults.conf file on the client.
- Add the following parameters to the bin/spark-submit command:
  --conf spark.kerberos.keytab=<keytab file path>--conf spark.kerberos.principal=<Principal account>
- Add the following parameter to the bin/spark-submit command:
  --keytab<keytab file path>--principal<Principal account>
Authentication by adding codes:
Authenticate in the application by obtaining principal and keytab files of the client.

Table 1 lists the authentication method for the sample code in security cluster environment.

**Table 1** Authentication methods
Sample Code	Mode	Authentication Method
sparknormal-examples	yarn-client	Command authentication, configuration authentication, or code authentication
sparknormal-examples	yarn-cluster	Either command authentication or configuration authentication
sparksecurity-examples (containing authentication code)	yarn-client	Code authentication
sparksecurity-examples (containing authentication code)	yarn-cluster	Not supported.

NOTE:

When using the yarn cluster mode in Spark projects, security authentication is not supported. Therefore, it is necessary to complete the security authentication before starting the application.
To ensure secure authentication in the Python sample project, you must configure the authentication parameter in the command used to run the application since the authentication codes are not provided.

Security Authentication Code (Java)

The safety authentication of the sample codes is completed by invoking the LoginUtil class. To learn more about the secure login process, refer to the chapter on security authentication APIs.

The Spark sample project code uses different authentication codes for different sample projects. These include basic safety authentication and basic safety authentication with ZooKeeper authentication. Table 2 describes the example authentication parameters used in the sample project. Change the parameter values based on the site requirements.

**Table 2** Parameter description
Parameter	Example Value	Description
userPrincipal	sparkuser	User Principal for authentication. Use the account prepared in Preparing MRS Application Development User.
userKeytabPath	/opt/FIclient/user.keytab	Keytab file used for authentication. Copy the user.keytab file of the prepared developer account to the directory indicated by the example parameter value.
ZKServerPrincipal	zookeeper/hadoop.<System domain name>	Principal of the ZooKeeper server. Contact the administrator to obtain the account.

The following code snippet belongs to the main method of the FemaleInfoCollection class in the com.huawei.bigdata.spark.examples package.

Basic safety authentication:

Spark Core and Spark SQL programs require only basic safety authentication codes since they do not need to access HBase or ZooKeeper. Add the following codes in the program and configure the safety authentication parameter as needed:

String userPrincipal = "sparkuser";
String userKeytabPath = "/opt/FIclient/user.keytab";
String krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf";
Configuration hadoopConf = new Configuration();
LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

ZooKeeper authentication:

Because the sample projects "Spark Streaming", "access Spark SQL with JDBC", and "Spark on HBase" require not only the basic safety authentication, but also the Principal of the ZooKeeper server to complete the safety authentication. Add the following codes in the program and configure the safety authentication parameter as needed:

String userPrincipal = "sparkuser";
String userKeytabPath = "/opt/FIclient/user.keytab";
String krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf";
String ZKServerPrincipal = "zookeeper/hadoop.<System domain name>";

String ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME = "Client";
String ZOOKEEPER_SERVER_PRINCIPAL_KEY = "zookeeper.server.principal";

Configuration hadoopConf = new Configuration();
LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userPrincipal, userKeytabPath);
LoginUtil.setZookeeperServerPrincipal(ZOOKEEPER_SERVER_PRINCIPAL_KEY, ZKServerPrincipal);
LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

Security Authentication Code (Scala)

The sample codes currently use the LoginUtil class for safety authentication. Refer to the chapter on unified authentication for information on secure login processes.

The Spark sample project code uses different authentication codes for different sample projects. These include basic safety authentication and basic safety authentication with ZooKeeper authentication. Table 3 describes the example authentication parameters used in the sample project. Change the parameter values based on the site requirements.

**Table 3** Parameter description
Parameter	Example Value	Description
userPrincipal	sparkuser	User Principal for authentication. Use the account prepared in Preparing MRS Application Development User.
userKeytabPath	/opt/FIclient/user.keytab	Keytab file used for authentication. Copy the user.keytab file of the prepared developer account to the directory indicated by the example parameter value.
ZKServerPrincipal	zookeeper/hadoop.<System domain name>	Principal of the ZooKeeper server. Contact the administrator to obtain the account.

Basic safety authentication:

val userPrincipal = "sparkuser"
val userKeytabPath = "/opt/FIclient/user.keytab"
val krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf"
val hadoopConf: Configuration  = new Configuration()
LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

ZooKeeper authentication:

val userPrincipal = "sparkuser"
val userKeytabPath = "/opt/FIclient/user.keytab"
val krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf"
val ZKServerPrincipal = "zookeeper/hadoop.<System domain name>"

val ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME: String = "Client"
val ZOOKEEPER_SERVER_PRINCIPAL_KEY: String = "zookeeper.server.principal"
val hadoopConf: Configuration  = new Configuration();
LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userPrincipal, userKeytabPath)
LoginUtil.setZookeeperServerPrincipal(ZOOKEEPER_SERVER_PRINCIPAL_KEY, ZKServerPrincipal)
LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

Parent topic: Preparing a Spark Application Development Environment

Previous topic: Preparing a Local Application Development Environment

Next topic: Importing and Configuring Spark Sample Projects