Configuring Security Authentication for Spark Applications

Overview

In a cluster with the security mode enabled, the components must be mutually authenticated before communicating with each other to ensure communication security.

When developing Spark applications, you may need Spark to communicate with Hadoop and HBase. Code for security authentication needs to be written into the Spark applications to ensure that the applications can work properly.

Three security authentication modes are available:

• Authentication by running command lines

  To authenticate, run this command in the Spark client before submitting the application for running or logging in to Spark SQL using the CLI:

  kinitComponent service user

• Authentication by configuring parameters:

  You can use any of the following methods to specify the security authentication information.

  • Configure the spark.kerberos.keytab and spark.kerberos.principal parameters in the spark-defaults.conf file on the client.
  • Add the following parameters to the bin/spark-submit command:

    --conf spark.kerberos.keytab=<keytab file path>--conf spark.kerberos.principal=<Principal account>

  • Add the following parameter to the bin/spark-submit command:

    --keytab<keytab file path>--principal<Principal account>

• Authentication by adding codes:

  Authenticate in the application by obtaining principal and keytab files of the client.

Table 1 lists the authentication method for the sample code in security cluster environment.

• When using the yarn cluster mode in Spark projects, security authentication is not supported. Therefore, it is necessary to complete the security authentication before starting the application.
• To ensure secure authentication in the Python sample project, you must configure the authentication parameter in the command used to run the application since the authentication codes are not provided.

Security Authentication Code (Java)

The safety authentication of the sample codes is completed by invoking the LoginUtil class. To learn more about the secure login process, refer to the chapter on security authentication APIs.

The Spark sample project code uses different authentication codes for different sample projects. These include basic safety authentication and basic safety authentication with ZooKeeper authentication. Table 2 describes the example authentication parameters used in the sample project. Change the parameter values based on the site requirements.

The following code snippet belongs to the main method of the FemaleInfoCollection class in the com.huawei.bigdata.spark.examples package.

• Basic safety authentication:
  Spark Core and Spark SQL programs require only basic safety authentication codes since they do not need to access HBase or ZooKeeper. Add the following codes in the program and configure the safety authentication parameter as needed:

  String userPrincipal = "sparkuser";
  String userKeytabPath = "/opt/FIclient/user.keytab";
  String krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf";
  Configuration hadoopConf = new Configuration();
  LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

• ZooKeeper authentication:

  Because the sample projects "Spark Streaming", "access Spark SQL with JDBC", and "Spark on HBase" require not only the basic safety authentication, but also the Principal of the ZooKeeper server to complete the safety authentication. Add the following codes in the program and configure the safety authentication parameter as needed:

  String userPrincipal = "sparkuser";
  String userKeytabPath = "/opt/FIclient/user.keytab";
  String krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf";
  String ZKServerPrincipal = "zookeeper/hadoop.<System domain name>";
  
  String ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME = "Client";
  String ZOOKEEPER_SERVER_PRINCIPAL_KEY = "zookeeper.server.principal";
  
  Configuration hadoopConf = new Configuration();
  LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userPrincipal, userKeytabPath);
  LoginUtil.setZookeeperServerPrincipal(ZOOKEEPER_SERVER_PRINCIPAL_KEY, ZKServerPrincipal);
  LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

Security Authentication Code (Scala)

The sample codes currently use the LoginUtil class for safety authentication. Refer to the chapter on unified authentication for information on secure login processes.

The Spark sample project code uses different authentication codes for different sample projects. These include basic safety authentication and basic safety authentication with ZooKeeper authentication. Table 3 describes the example authentication parameters used in the sample project. Change the parameter values based on the site requirements.

• Basic safety authentication:
  Spark Core and Spark SQL programs require only basic safety authentication codes since they do not need to access HBase or ZooKeeper. Add the following codes in the program and configure the safety authentication parameter as needed:

  val userPrincipal = "sparkuser"
  val userKeytabPath = "/opt/FIclient/user.keytab"
  val krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf"
  val hadoopConf: Configuration  = new Configuration()
  LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);

• ZooKeeper authentication:

  Because the sample projects "Spark Streaming", "access Spark SQL with JDBC", and "Spark on HBase" require not only the basic safety authentication, but also the Principal of the ZooKeeper server to complete the safety authentication. Add the following codes in the program and configure the safety authentication parameter as needed:

  val userPrincipal = "sparkuser"
  val userKeytabPath = "/opt/FIclient/user.keytab"
  val krb5ConfPath = "/opt/FIclient/KrbClient/kerberos/var/krb5kdc/krb5.conf"
  val ZKServerPrincipal = "zookeeper/hadoop.<System domain name>"
  
  val ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME: String = "Client"
  val ZOOKEEPER_SERVER_PRINCIPAL_KEY: String = "zookeeper.server.principal"
  val hadoopConf: Configuration  = new Configuration();
  LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userPrincipal, userKeytabPath)
  LoginUtil.setZookeeperServerPrincipal(ZOOKEEPER_SERVER_PRINCIPAL_KEY, ZKServerPrincipal)
  LoginUtil.login(userPrincipal, userKeytabPath, krb5ConfPath, hadoopConf);