On this page

Show all

Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Flink Development Guide/ Preparing a Flink Application Development Environment/ Preparing the Flink Application Security Authentication

Preparing the Flink Application Security Authentication

Updated on 2024-08-16 GMT+08:00

View PDF

If Kerberos authentication is enabled for the MRS cluster, perform the following steps to prepare a development user. If Kerberos authentication is not enabled, skip the following steps.

In a security cluster environment, the components must be mutually authenticated before communicating with each other to ensure communication security.

When submitting a Flink application, you need to communicate with Yarn and HDFS. Security authentication needs to be configured for the Flink application to be submitted to ensure that the Flink application can work properly.

Flink supports authentication and encrypted transmission. This section describes preparations required for using authentication and encrypted transmission.

Security Authentication

Flink uses the following two authentication modes:

Kerberos authentication: It is used between the Flink YARN client and YARN ResourceManager, JobManager and ZooKeeper, JobManager and HDFS, TaskManager and HDFS, Kafka and TaskManager, as well as TaskManager and ZooKeeper.

Internal authentication mechanism of Yarn: It is used between Yarn ResourceManager and ApplicationMaster.

NOTE:

Flink JobManager and YARN ApplicationMaster are in the same process.

**Table 1** Security authentication mode
Security Authentication Mode	Description	Configuration
Kerberos authentication	Currently, only keytab authentication is supported.	Download the user keytab file from the KDC server, and place the keytab file to a folder on the host of the Flink client (for example, /home/flinkuser/keytab). Configure the following parameters in the ${FLINK_HOME}/conf/flink-conf.yaml file: Keytab file path security.kerberos.login.keytab: /home/flinkuser/keytab/user.keytab NOTE: /home/flinkuser/keytab/ indicates the directory for storing the keytab file. Principal name (developer username). security.kerberos.login.principal:flinkuser In HA mode, if Zookeeper is configured, ZooKeeper Kerberos authentication must be configured as follows: zookeeper.sasl.disable: false security.kerberos.login.contexts: Client If Kerberos authentication is required between the Kafka client and Kafka broker, configure it as follows: security.kerberos.login.contexts: Client,KafkaClient
Internal authentication of YARN	The user does not need to configure this internal authentication mode.	-

NOTE:

One Flink cluster belongs to only one user. One user can create multiple Flink clusters.

Encrypted Transmission

Flink uses the following three encrypted transmission modes:

Encrypted transmission inside YARN: It is used between the Flink YARN client and YARN ResourceManager, as well as YARN ResourceManager and JobManager.
SSL transmission: It is used between the Flink YARN client and JobManager, JobManager and TaskManager, as well as TaskManagers.
Encrypted transmission inside Hadoop: It is used between JobManager and HDFS, TaskManager and HDFS, JobManager and ZooKeeper, and TaskManager and ZooKeeper.
NOTE:

You do not need to configure encryption inside YARN and Hadoop, but need to configure SSL transmission.

To configure SSL transmission, configure the flink-conf.yaml file on the client.

Turn on the SSL switch and set SSL encryption algorithms. Table 2 describes the parameters. Set the parameters based on site requirements.

**Table 2** Parameters
Parameter	Example Value	Description
security.ssl.internal.enabled	true	Switch to enable internal SSL
akka.ssl.enabled	true	Switch to enable Akka SSL
blob.service.ssl.enabled	true	Switch to enable SSL of the BLOB channels
taskmanager.data.ssl.enabled	true	Switch to enable SSL for communications between TaskManagers
security.ssl.algorithms	TLS_RSA_WITH_AES_128_CBC_SHA256	SSL encryption algorithms

The following parameters are not included in the default Flink configurations of MRS. You can add them if necessary. If you enable SSL for external connections, the proxy of YARN cannot access the Flink page. This is because Yarn does not support the HTTPS proxy. There can be security risks if a configuration file contains the authentication password. You are advised to delete the configuration file or use other secure methods to keep the password.

Parameter	Example Value	Description
security.ssl.rest.enabled	true	Switch to enable external SSL
security.ssl.rest.keystore	${path}/flink.keystore	Path for storing the keystore
security.ssl.rest.keystore-password	123456	Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.rest.key-password	123456	Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.rest.truststore	${path}/flink.truststore	Path for storing the truststore
security.ssl.rest.truststore-password	123456	Password of the truststore. The value 123456 indicates a user-defined password.

NOTE:

Enabling SSL for data transmission between TaskManagers may pose great impact on system performance. You need to take both security and performance into consideration.

In the bin directory of the Flink client, run the sh generate_keystore.sh <Password> command. The configuration items in Table 3 are set by default. You can also set the configuration items yourself. There can be security risks if a command contains the authentication password. You are advised to disable the command recording function (history) before running the command.

**Table 3** Parameters
Parameter	Example Value	Description
security.ssl.internal.keystore	${path}/flink.keystore	Path for storing the keystore file. flink.keystore indicates the name of the keystore file generated by the generate_keystore.sh* tool.
security.ssl.internal.keystore-password	123456	Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.internal.key-password	123456	Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.internal.truststore	${path}/flink.truststore	Path for storing the truststore file. flink.truststore indicates the name of the truststore file generated by the generate_keystore.sh* tool.
security.ssl.internal.truststore-password	123456	Password of the truststore. The value 123456 indicates a user-defined password.

If SSL for external connections is enabled, that is, security.ssl.rest.enabled is set to true, the following parameters need to be set:

Parameter	Example Value	Description
security.ssl.rest.keystore	${path}/flink.keystore	Path for storing the keystore
security.ssl.rest.keystore-password	123456	Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.rest.key-password	123456	Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.rest.truststore	${path}/flink.truststore	Path for storing the truststore
security.ssl.rest.truststore-password	123456	Password of the truststore. The value 123456 indicates a user-defined password.

path indicates a user-defined directory that is used to store configuration files of the SSL keystore and truststore. The commands vary according to the relative path and absolute path. The details are as follows:

NOTE:

Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path.
```
security.ssl.internal.keystore: ssl/flink.keystore
security.ssl.internal.truststore: ssl/flink.truststore
```

If the keystore or truststore file path is an absolute path, the keystore or truststore file must exist in the absolute path on Flink Client and all nodes.

security.ssl.internal.keystore: /opt/client/Flink/flink/conf/flink.keystore
security.ssl.internal.truststore: /opt/client/Flink/flink/conf/flink.truststore

Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path. Flink can transfer the keystore and truststore files using either of the following methods:
- Add the -t option to the CLI yarn-session.sh command of Flink to transfer the keystore and truststore files to execution nodes. Example:
```
./bin/yarn-session.sh -t ssl/ -n 2
```
- Add the -yt option to the flink run command to transfer the keystore and truststore files to execution nodes. Example:
```
./bin/flink run -yt ssl/ -ys 3 -yn 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000
```
  NOTE:
  - In the preceding example, ssl/ is the sub-directory of the Flink client directory and is used to store configuration files of the SSL keystore and truststore.
  - The relative path of ssl/ must be accessible from the current path where the Flink client command is executed.
If the keystore or truststore file path is an absolute path, the keystore and truststore files must exist in the absolute path on Flink Client and all nodes. In addition, the user who submits the job must have permission to read the files.
Either of the following methods can be used to run applications. The -t or -yt option does not need to be added to transfer the keystore and truststore files.
- Run the CLI yarn-session.sh command of Flink to execute applications. Example:
```
./bin/yarn-session.sh -n 2
```
- Run the flink run command to execute applications. Example:
```
./bin/flink run  -ys 3 -yn 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000
```

Parent topic: Preparing a Flink Application Development Environment

Previous topic: (Optional) Creating Flink Sample Projects

Next topic: Developing Flink Applications

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot