Preparing the Flink Application Security Authentication
If Kerberos authentication is enabled for the MRS cluster, perform the following steps to prepare a development user. If Kerberos authentication is not enabled, skip the following steps.
In a security cluster environment, the components must be mutually authenticated before communicating with each other to ensure communication security.
When submitting a Flink application, you need to communicate with Yarn and HDFS. Security authentication needs to be configured for the Flink application to be submitted to ensure that the Flink application can work properly.
Flink supports authentication and encrypted transmission. This section describes preparations required for using authentication and encrypted transmission.
Security Authentication
Flink uses the following two authentication modes:
- Kerberos authentication: It is used between the Flink YARN client and YARN ResourceManager, JobManager and ZooKeeper, JobManager and HDFS, TaskManager and HDFS, Kafka and TaskManager, as well as TaskManager and ZooKeeper.
- Internal authentication mechanism of Yarn: It is used between Yarn ResourceManager and ApplicationMaster.
Flink JobManager and YARN ApplicationMaster are in the same process.
Table 1 Security authentication mode Security Authentication Mode
Description
Configuration
Kerberos authentication
Currently, only keytab authentication is supported.
- Download the user keytab file from the KDC server, and place the keytab file to a folder on the host of the Flink client (for example, /home/flinkuser/keytab).
- Configure the following parameters in the ${FLINK_HOME}/conf/flink-conf.yaml file:
- Keytab file path
security.kerberos.login.keytab: /home/flinkuser/keytab/user.keytab
NOTE:/home/flinkuser/keytab/ indicates the directory for storing the keytab file.
- Principal name (developer username).
security.kerberos.login.principal:flinkuser
- In HA mode, if Zookeeper is configured, ZooKeeper Kerberos authentication must be configured as follows:
zookeeper.sasl.disable: false security.kerberos.login.contexts: Client
- If Kerberos authentication is required between the Kafka client and Kafka broker, configure it as follows:
security.kerberos.login.contexts: Client,KafkaClient
- Keytab file path
Internal authentication of YARN
The user does not need to configure this internal authentication mode.
-
One Flink cluster belongs to only one user. One user can create multiple Flink clusters.
Encrypted Transmission
Flink uses the following three encrypted transmission modes:
- Encrypted transmission inside YARN: It is used between the Flink YARN client and YARN ResourceManager, as well as YARN ResourceManager and JobManager.
- SSL transmission: It is used between the Flink YARN client and JobManager, JobManager and TaskManager, as well as TaskManagers.
- Encrypted transmission inside Hadoop: It is used between JobManager and HDFS, TaskManager and HDFS, JobManager and ZooKeeper, and TaskManager and ZooKeeper.
You do not need to configure encryption inside YARN and Hadoop, but need to configure SSL transmission.
To configure SSL transmission, configure the flink-conf.yaml file on the client.
- Turn on the SSL switch and set SSL encryption algorithms. Table 2 describes the parameters. Set the parameters based on site requirements.
Table 2 Parameters Parameter
Example Value
Description
security.ssl.internal.enabled
true
Switch to enable internal SSL
akka.ssl.enabled
true
Switch to enable Akka SSL
blob.service.ssl.enabled
true
Switch to enable SSL of the BLOB channels
taskmanager.data.ssl.enabled
true
Switch to enable SSL for communications between TaskManagers
security.ssl.algorithms
TLS_RSA_WITH_AES_128_CBC_SHA256
SSL encryption algorithms
The following parameters are not included in the default Flink configurations of MRS. You can add them if necessary. If you enable SSL for external connections, the proxy of YARN cannot access the Flink page. This is because Yarn does not support the HTTPS proxy. There can be security risks if a configuration file contains the authentication password. You are advised to delete the configuration file or use other secure methods to keep the password.
Parameter
Example Value
Description
security.ssl.rest.enabled
true
Switch to enable external SSL
security.ssl.rest.keystore
${path}/flink.keystore
Path for storing the keystore
security.ssl.rest.keystore-password
123456
Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.rest.key-password
123456
Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.rest.truststore
${path}/flink.truststore
Path for storing the truststore
security.ssl.rest.truststore-password
123456
Password of the truststore. The value 123456 indicates a user-defined password.
Enabling SSL for data transmission between TaskManagers may pose great impact on system performance. You need to take both security and performance into consideration.
- In the bin directory of the Flink client, run the sh generate_keystore.sh <Password> command. The configuration items in Table 3 are set by default. You can also set the configuration items yourself. There can be security risks if a command contains the authentication password. You are advised to disable the command recording function (history) before running the command.
Table 3 Parameters Parameter
Example Value
Description
security.ssl.internal.keystore
${path}/flink.keystore
Path for storing the keystore file. flink.keystore indicates the name of the keystore file generated by the generate_keystore.sh* tool.
security.ssl.internal.keystore-password
123456
Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.internal.key-password
123456
Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.internal.truststore
${path}/flink.truststore
Path for storing the truststore file. flink.truststore indicates the name of the truststore file generated by the generate_keystore.sh* tool.
security.ssl.internal.truststore-password
123456
Password of the truststore. The value 123456 indicates a user-defined password.
If SSL for external connections is enabled, that is, security.ssl.rest.enabled is set to true, the following parameters need to be set:
Parameter
Example Value
Description
security.ssl.rest.keystore
${path}/flink.keystore
Path for storing the keystore
security.ssl.rest.keystore-password
123456
Password of the keystore. The value 123456 indicates a user-defined password.
security.ssl.rest.key-password
123456
Password of the SSL key. The value 123456 indicates a user-defined password.
security.ssl.rest.truststore
${path}/flink.truststore
Path for storing the truststore
security.ssl.rest.truststore-password
123456
Password of the truststore. The value 123456 indicates a user-defined password.
path indicates a user-defined directory that is used to store configuration files of the SSL keystore and truststore. The commands vary according to the relative path and absolute path. The details are as follows:
- Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path.
security.ssl.internal.keystore: ssl/flink.keystore security.ssl.internal.truststore: ssl/flink.truststore
- If the keystore or truststore file path is an absolute path, the keystore or truststore file must exist in the absolute path on Flink Client and all nodes.
security.ssl.internal.keystore: /opt/client/Flink/flink/conf/flink.keystore security.ssl.internal.truststore: /opt/client/Flink/flink/conf/flink.truststore
- Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path. Flink can transfer the keystore and truststore files using either of the following methods:
- Add the -t option to the CLI yarn-session.sh command of Flink to transfer the keystore and truststore files to execution nodes. Example:
./bin/yarn-session.sh -t ssl/ -n 2
- Add the -yt option to the flink run command to transfer the keystore and truststore files to execution nodes. Example:
./bin/flink run -yt ssl/ -ys 3 -yn 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000
- In the preceding example, ssl/ is the sub-directory of the Flink client directory and is used to store configuration files of the SSL keystore and truststore.
- The relative path of ssl/ must be accessible from the current path where the Flink client command is executed.
- Add the -t option to the CLI yarn-session.sh command of Flink to transfer the keystore and truststore files to execution nodes. Example:
- If the keystore or truststore file path is an absolute path, the keystore and truststore files must exist in the absolute path on Flink Client and all nodes. In addition, the user who submits the job must have permission to read the files.
Either of the following methods can be used to run applications. The -t or -yt option does not need to be added to transfer the keystore and truststore files.
- Run the CLI yarn-session.sh command of Flink to execute applications. Example:
./bin/yarn-session.sh -n 2
- Run the flink run command to execute applications. Example:
./bin/flink run -ys 3 -yn 3 -m yarn-cluster -c com.huawei.SocketWindowWordCount lib/flink-eg-1.0.jar --hostname r3-d3 --port 9000
- Run the CLI yarn-session.sh command of Flink to execute applications. Example:
- Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
