Preparing a Spark Application Development User

Prerequisites

Kerberos authentication has been enabled for the MRS cluster. Skip this step if Kerberos authentication is not enabled for the cluster.

Scenario

The development user is used to run the sample project. The user must have HDFS, Yarn, and Hive permissions to run the Spark sample project.

Procedure

1. Log in to MRS Manager. For details, see Logging in to MRS Manager.
2. On MRS Manager, choose System > Manage Role > Create Role.
   1. Enter a role name, for example, sparkrole.
   2. In Permission, choose HBase > HBase Scope > global. Select Create for default.
   3. In Permission, choose HBase > HBase Scope > global > hbase. Select Execute for hbase:meta.
   4. Modify the role. In Permission, choose HDFS > File System, select Read, Write, and Execute.
   5. In Permission, select HDFS > File System > hdfs://hacluster/ > user > hive, and select Execute.
   6. In Permission, choose HDFS > File System > hdfs://hacluster/ > user > hive > warehouse, and select Read, Write, and Execute.
   7. In Permission, choose Hive > Hive Read Write Privileges and select Create for default.
   8. In Permission, choose Yarn > Scheduler Queue > root, and select Submit for default.
   9. Click OK.

3. On MRS Manager, choose System > Manage User > Create User to create a user for the sample project. Enter a username, for example, sparkuser. Set User Type to Machine-machine, and select both supergroup and kafkaadmin in User Group. Set Primary Group to supergroup, select the sparkrole role to obtain permissions, and click OK.
   

   Users using the Spark Streaming program need the kafkaadmin group permission to operate the Kafka component.

4. On MRS Manager, choose System > Manage User and select sparkuser. Click to download an authentication credential file. Save the file and decompress it to obtain the keytab and krb5.conf files. They are used for security authentication in the sample project. For details how to use them, see Preparing the Authentication Mechanism Code.