Flink Troubleshooting

Running the yarn-session Command Fails to Create a Flink Cluster When a Different User Is Used

There are two users with the same permissions: testuser and bdpuser. When user testuser is used to create a Flink cluster, no error message is displayed. While user bdpuser is used to create a Flink cluster, an error message is displayed during the yarn-session.sh command execution.

2019-01-02 14:28:09,098 | ERROR | [main] | Ensure path threw exception | org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:566) 
org.apache.flink.shaded.zookeeper.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /flink/application_1545397824912_0022

This is because that the HA configuration item is not modified. In the Flink configuration file, the default value of high-availability.zookeeper.client.acl is creator, indicating that only the creator has the access permission. A new user cannot access the directory on ZooKeeper. As a result, the yarn-session.sh command execution fails.

Perform the following steps to solve the problem:

Modify the value of high-availability.zookeeper.path.root in the conf/flink-conf.yaml file. For example, run the following command:
```
high-availability.zookeeper.path.root: flink2
```
Submit the Flink job again.

Error Message "security.kerberos.login.keytab" Is Displayed When a Command Is Executed on the Flink Client

The client was successfully installed. When you run a client command, for example, yarn-session.sh, an error message is displayed, as shown in the following figure.

[root@host01 bin]# yarn-session.sh
2018-10-25 01:22:06,454 | ERROR | [main] | Error while trying to split key and value in configuration file /opt/flinkclient/Flink/flink/conf/flink-conf.yaml:80: "security.kerberos.login.keytab: " | org.apache.flink.configuration.GlobalConfiguration (GlobalConfiguration.java:160)
Exception in thread "main" org.apache.flink.configuration.IllegalConfigurationException: Error while parsing YAML configuration file :80: "security.kerberos.login.keytab: "

In a secure cluster environment, Flink requires security authentication. The security authentication is not configured on the current client.

Flink uses the following two authentication modes:
- Kerberos authentication: used by Flink yarn client, Yarn Resource Manager, JobManager, HDFS, TaskManager, Kafka, and ZooKeeper
- Internal authentication mechanism of Yarn: used between Yarn ResourceManager and ApplicationMaster
If security mode is enabled for a cluster, you must use the Kerberos authentication and security cookie authentication. As shown in the logs, it is found that the security.kerberos.login.keytab setting in the configuration file is incorrect and the security configuration is not performed.

Perform the following steps to solve the problem:

Download the user keytab file from MRS and save it to a directory on the node where the Flink client is deployed.
Configure the following parameters in the flink-conf.yaml file:
1. Keytab path
```
security.kerberos.login.keytab: /home/flinkuser/keytab/abc222.keytab
```
  - /home/flinkuser/keytab/abc222.keytab indicates the user directory storing the keytab file in 1.
  - You need to ensure that the client user has the permission on the corresponding directory.
1. Principal name
```
security.kerberos.login.principal: abc222
```
1. In HA mode, if ZooKeeper is configured, the Kerberos authentication configuration items must be configured.
```
zookeeper.sasl.disable: false 
security.kerberos.login.contexts: Client
```
1. If you want to perform Kerberos authentication between Kafka client and Kafka broker, set the value as follows:
```
security.kerberos.login.contexts: Client,KafkaClient
```