Preparing for Security Authentication

Scenarios

In secure cluster mode, all clusters must authenticate each other before communicating.

When you submit Flink applications, Flink applications need to communicate with components such as YARN and HDFS. Therefore, security authentication must be configured for Flink.

Flink supports authentication and encrypted transmission. This section describes preparations required for authentication and encrypted transmission.

Authentication

Figure 1 Authentication modes of Flink

Flink uses following authentication modes:

Kerberos authentication: Kerberos authentication is used between Flink YARN client and YARN ResourceManager, JobManager and ZooKeeper, JobManager and HDFS, TaskManager and HDFS, Kafka and TaskManager, as well as TaskManager and ZooKeeper.
Security cookie authentication: Security cookie authentication is used between Flink YARN client and JobManager, JobManager and TaskManager, as well as TaskManager and TaskManager.
Authentication inside YARN: The Internal authentication mechanism of YARN is used between YARN ResourceManager and ApplicationMaster (AM).
- Flink JobManager and YARN ApplicationMaster are in the same process.
- If you want to use the security mode, the kerberos authentication and security cookie authentication are mandatory.

**Table 1** Authentication modes
Authentication Mode	Configuration Method
Kerberos authentication Currently, only keytab authentication mode is supported.	Download user keytab from FusionInsight Manager, and place the keytab to a directory on the host of Flink client. Configure following parameters in the flink-conf.yaml file: Add the service IP address of the node where the client is installed and IP address of the master node to the jobmanager.web.access-control-allow-origin and jobmanager.web.allow-access-address configuration items in the /opt/hadoopclient/Flink/flink/conf/flink-conf.yaml file. Use commas (,) to separate the IP addresses. jobmanager.web.access-control-allow-origin: xx.xx.xxx.xxx,xx.xx.xxx.xxx,xx.xx.xxx.xxx jobmanager.web.allow-access-address: xx.xx.xxx.xxx,xx.xx.xxx.xxx,xx.xx.xxx.xxx NOTE: Node outside the cluster: IP address of the ECS where the client is installed.Node inside the cluster: In the navigation tree of the MRS management console, choose Clusters > Active Clusters, select a cluster, and click its name to switch to the cluster details page.On the Nodes tab page, view the IP address of the node where the client is installed. Keytab path. security.kerberos.login.keytab: /home/flinkuser/keytab/flinkuser.keytab NOTE: /home/flinkuser/keytab/ indicates the directory for storing keytab. Principal name. security.kerberos.login.principal. flinkuser In HA mode, if Zookeeper is configured, the Kerberos authentication configuration items must be configured as follows: zookeeper.sasl.disable: false security.kerberos.login.contexts: Client If you want to perform Kerberos authentication between Kafka client and Kafka broker, set the value as follows: security.kerberos.login.contexts: Client,KafkaClient
Security cookie authentication	Obtain the SSL certificate and save it to the Flink client. For details, see Authentication and Encryption. Assign values to the following configuration items in the flink-conf.yaml file in the conf directory of the Flink client: Set the security.ssl.key-password, security.ssl.keystore-password, and security.ssl.truststore-password to <password>. Set security.ssl.keystore to the relative path of the keystore file, that is, ssl/flink.keystore. Set security.ssl.truststore to the relative path of the truststore file, that is, ssl/flink.truststore. Set security.cookie to a random password string. NOTE: The generated flink.keystore, flink.truststore, and security.cookie items are automatically filled in the corresponding configuration items in flink-conf.yaml as shown in Authentication and Encryption. The values of security.ssl.key-password, security.ssl.keystore-password, and security.ssl.truststore-password need to be obtained using the Manager plaintext encryption API by running the curl -k -i -u username:password -X POST -HContent-type:application/json -d '{"plainText":"password"}' 'https://x.x.x.x:28443/web/api/v2/tools/encrypt'.In the preceding command, Username:Password indicates the user name and password for logging in to the system. The password if "plainText" indicates the one used to call the generate_keystore.sh script. x.x.x.x indicates the floating IP address of Manager. Set security.enable: true to true and configure security cookie. Example: security.cookie: ae70acc9-9795-4c48-ad35-8b5adc8071744f605d1d-2726-432e-88ae-dd39bfec40a9
Authentication inside YARN	The authentication mode does not need to be configured.

One Flink cluster supports only one user. One user can create multiple Flink clusters.

Encrypted Transmission

Figure 2 Encrypted transmission of Flink

Flink uses following encrypted transmission modes:

Encrypted transmission inside YARN: encrypted transmission is used between Flink YARN client and YARN ResourceManager, as well as YARN ResourceManager and JobManager.
SSL transmission: SSL transmission is used between Flink YARN client and JobManager, JobManager and TaskManager, as well as TaskManager and TaskManager.
Encrypted transmission inside Hadoop: The internal encrypted transmission mode of Hadoop used between JobManager and HDFS, TaskManager and HDFS, JobManager and ZooKeeper, as well as TaskManager and ZooKeeper.

Configuration about SSL encrypted transmission is mandatory while configuration about encryption of YARN and Hadoop is not required.

In the flink-conf.yaml file on the client, configure following parameters to configure the SSL transmission.

Enable SSL and configure the SSL encryption algorithm. Table 2 lists the parameters. Modify the parameter value as required.

**Table 2** Parameter Description
Parameter	Example Parameter Value	Description
security.ssl.enabled	true	Enable SSL
akka.ssl.enabled	true	Enable Akka SSL
blob.service.ssl.enabled	true	Enable SSL for blob channel
taskmanager.data.ssl.enabled	true	Enable SSL for TaskManagers
security.ssl.algorithms	TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384	Configure the SSL encryption algorithm

Enabling SSL transmission between TaskManagers may pose great impact on system performance.

In the bin directory on the Flink client, run the sh generate_keystore.sh <password> command.see Authentication and Encryption.The configuration items in Table 3 are set by default. You can also configure them manually.

**Table 3** Parameter Description
Parameter	Example Parameter Value	Description
security.ssl.keystore	${path}/flink.keystore	Path for storing keystore. flink.keystore indicates the name of the keystore file generated by the generate_keystore.sh* tool.
security.ssl.keystore-password	123456	Password of keystore. The 123456 indicates the user-defined password.
security.ssl.key-password	123456	Password of SSL key. The 123456 indicates the user-defined password.
security.ssl.truststore	${path}/flink.truststore	Path for storing the truststore. flink.truststore indicates the name of the truststore file generated by the generate_keystore.sh* tool.
security.ssl.truststore-password	123456	Password of truststore, The 123456 indicates the user-defined password.

The path directory is used to store SSL configuration files of SSL keystore and truststore. This directory is user-defined. Commands for absolute path and relative path are different. For details, see 3 and 4.

If the keystore or truststore file path is a relative path, either of the following method can be used to transmit the keystore and truststore file:
- Add - t option to the CLI yarn-session.sh command. For example:
  cd /opt/client/Flink/flink
  
  ./bin/yarn-session.sh -t ssl/
- Add -yt option to the flink run command. For example:
  ./bin/flink run -yt ssl/ -ys 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount ../lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000
  - The ssl/ directory is used to store SSL configuration files of SSL keystore and truststore.
  - The relative path ssl/ of current path where the Flink Client command is run must be accessible.
If the keystore or truststore file path is an absolute path, the keystore or truststore file must exist in the absolute path on Flink Client and all the YARN nodes.
Either of the following methods can be used to run applications. The -t or -yt option does not need to be added to transmit the keystore and truststore file.
- Run the CLI yarn-session.sh command of Flink to execute applications.
  ./bin/yarn-session.sh
- Run the flink run command to execute applications.
  ./bin/flink run -ys 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount ../lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000