Preparing a Flink Application Development User

Updated on 2024-12-10 GMT+08:00

View PDF

The development user is used to run the sample project. In a security cluster, only users with the permissions on HDFS, YARN, Kafka, and Flink are allowed to run Flink sample projects.

Prerequisites

Kerberos authentication has been enabled for the MRS cluster. Skip this step if Kerberos authentication is not enabled for the cluster.

Procedure

Log in to MRS Manager and choose System > Manage Role > Create Role.
1. Enter a role name, for example, flinkrole.
2. In Permission, choose HDFS > File System > hdfs://hacluster/ and select Read, Write, and Execute. After you finish configuring this service, click Service in the Permission area.
3. In Permission, choose Yarn > Scheduler Queue > root. Select Submit for default, and click OK.
  NOTE:
  After you submit applications, WARN logs are printed on the client based on your configuration about the preceding role. The WARN log is generated because Flink obtains the remaining resource value from YARN for detection and evaluation. However, the operation requires the admin permission, which is not granted to you. Ignore the WARN log because it does not affect the job submission. Content of the WARN log is as follows:
  Get node resource from yarn cluster. Yarn cluster occur exception: org.apache.hadoop.yarn.exceptions.YarnPermissionDeniedException: User flinkuser does not have privilege to see, admin only
On MRS Manager, choose System > Manage User Group > Create User Group to create a user group for the sample project, for example, flinkgroup.
On MRS Manager, choose System > Manage User > Create User to create a user for the sample project. Enter a username, for example, flinkuser. Set User Type to Human-machine, and select flinkgroup and hadoop in User Group. Select flinkrole in Assign Rights by Role, and click OK.
NOTE:
- You can use this user only after changing the password of user flinkuser on the client.
- If a user wants to interconnect with Kafka, a hybrid cluster with Flink and Kafka components is required, or cross-cluster mutual trust needs to be configured for the cluster with Flink and the cluster with Kafka components. Additionally, user flinkuser is added to the kafkaadmin user group.
- If a user wants to run a sample project (in Scala or Java) of an application of producing and consuming data in Kafka, the user needs to be added to the kafkaadmin group.
On MRS Manager, choose System > Manage User and select flinkuser. Download an authentication credential file, save the file and decompress it to obtain the keytab and krb5.conf files, and copy the krb5.conf file to the /etc directory of the client. They are used for security authentication in the sample project. For details how to use them, see Preparing the Flink Application Security Authentication.