Help Center > > User Guide> MRS Cluster Component Operation Guide> Using Flink> Security Hardening

Security Hardening

Updated at: Feb 12, 2020 GMT+08:00

Authentication and Encryption

Security authentication

Flink uses the following two authentication modes:

  • Kerberos authentication: It is used between the Flink YARN client and YARN ResourceManager, JobManager and ZooKeeper, JobManager and HDFS, TaskManager and HDFS, Kafka and TaskManager, as well as TaskManager and ZooKeeper.
  • Internal authentication mechanism of YARN: It is used between YARN ResourceManager and ApplicationMaster.
    • Flink JobManager and YARN ApplicationMaster are in the same process.
    • If Kerberos authentication is enabled for the user's cluster, Kerberos authentication is required.
    Table 1 Security authentication modes

    Security Authentication Mode

    Description

    Configuration

    Kerberos authentication

    Currently, only keytab authentication is supported.

    1. Download the user keytab from the KDC server, and place the keytab to a folder on the host of the Flink client.
    2. Configure following parameters in the flink-conf.yaml file:
      1. Keytab file path
        security.kerberos.login.keytab: /home/flinkuser/keytab/abc222.keytab

        Note:

        /home/flinkuser/keytab/abc222.keytab indicates the user directory.

      2. Principal name
        security.kerberos.login.principal: abc222
      3. In HA mode, if Zookeeper is configured, ZooKeeper Kerberos authentication must be configured as follows:
        zookeeper.sasl.disable: false
        security.kerberos.login.contexts: Client
      4. If Kerberos authentication is required between the Kafka client and Kafka broker, configure it as follows:
        security.kerberos.login.contexts: Client,KafkaClient

    Internal authentication of YARN

    The user does not need to configure this internal authentication mode.

    -

    One Flink cluster belongs to only one user. One user can create multiple Flink clusters.

    Encrypted transmission

    Flink uses the following three encrypted transmission modes:

    • Encrypted transmission inside YARN: It is used between the Flink YARN client and YARN ResourceManager, as well as YARN ResourceManager and JobManager.
    • SSL transmission: It is used between the Flink YARN client and JobManager, JobManager and TaskManager, as well as TaskManagers.
    • Encrypted transmission inside Hadoop: It is used between JobManager and HDFS, TaskManager and HDFS, JobManager and ZooKeeper, and TaskManager and ZooKeeper.

    You do not need to configure encryption inside YARN and Hadoop, but need to configure SSL transmission.

To configure SSL transmission, configure the flink-conf.yaml file on the client:

  1. Turn on the SSL switch and set SSL encryption algorithms. Table 2 describes the parameters. Set the parameters based on site requirements.

    Table 2 Parameter description

    Parameter

    Example Value

    Description

    security.ssl.internal.enabled

    true

    Switch to enable internal SSL

    akka.ssl.enabled

    true

    Switch to enable Akka SSL

    blob.service.ssl.enabled

    true

    Switch to enable SSL of the BLOB channels

    taskmanager.data.ssl.enabled

    true

    Switch to enable SSL for communications between TaskManagers

    security.ssl.algorithms

    TLS_RSA_WITH_AES128CBC_SHA256

    SSL encryption algorithms

    The following parameters do not exist in the default Flink configuration of MRS. If you want to enable SSL for external connections, you need to add the following parameters. After SSL for external connection is enabled, the native Flink page cannot be accessed using a YARN proxy, because the YARN open-source version cannot process HTTPS requests using a proxy. However, you can create a Windows VM in the same VPC of the cluster and access the native Flink page from the VM.
    Table 3 Parameter description

    Parameter

    Example Value

    Description

    security.ssl.rest.enabled

    true

    Switch to enable external SSL. If this parameter is set to true, set the related parameters by referring to Table 5.

    security.ssl.rest.keystore

    ${path}/flink.keystore

    Path for storing the keystore

    security.ssl.rest.keystore-password

    123456

    Password of the keystore. The value 123456 indicates a user-defined password.

    security.ssl.rest.key-password

    123456

    Password of the SSL key. The value 123456 indicates a user-defined password.

    security.ssl.rest.truststore

    ${path}/flink.truststore

    Path for storing the truststore

    security.ssl.rest.truststore-password

    123456

    Password of the truststore. The value 123456 indicates a user-defined password.

    Enabling SSL for data transmission between TaskManagers may pose great impact on system performance. You need to take both security and performance into consideration.

  2. In the bin directory of the Flink client, run the sh generate_keystore.sh <password> command. The configuration items in Table 4 are set by default. You can also set the configuration items manually.

    Table 4 Parameter description

    Parameter

    Example Value

    Description

    security.ssl.internal.keystore

    ${path}/flink.keystore

    Path for storing the keystore file. flink.keystore indicates the name of the keystore file generated by the generate_keystore.sh* tool.

    security.ssl.internal.keystore-password

    123456

    Password of the keystore. The value 123456 indicates a user-defined password.

    security.ssl.internal.key-password

    123456

    Password of the SSL key. The value 123456 indicates a user-defined password.

    security.ssl.internal.truststore

    ${path}/flink.truststore

    Path for storing the truststore file. flink.truststore indicates the name of the truststore file generated by the generate_keystore.sh* tool.

    security.ssl.internal.truststore-password

    123456

    Password of the truststore. The value 123456 indicates a user-defined password.

    If SSL for external connections is enabled, that is, security.ssl.rest.enabled is set to true, the following parameters need to be set:

    Table 5 Parameter description

    Parameter

    Example Value

    Description

    security.ssl.rest.enabled

    true

    Switch to enable external SSL. If this parameter is set to true, set the related parameters by referring to Table 5.

    security.ssl.rest.keystore

    ${path}/flink.keystore

    Path for storing the keystore

    security.ssl.rest.keystore-password

    123456

    Password of the keystore. The value 123456 indicates a user-defined password.

    security.ssl.rest.key-password

    123456

    Password of the SSL key. The value 123456 indicates a user-defined password.

    security.ssl.rest.truststore

    ${path}/flink.truststore

    Path for storing the truststore

    security.ssl.rest.truststore-password

    123456

    Password of the truststore. The value 123456 indicates a user-defined password.

    path indicates a user-defined directory that is used to store configuration files of the SSL keystore and truststore. The commands vary according to the relative path and absolute path. The details are as follows:

    • Configure the file path storing the keystore or truststore file to a relative path, and the Flink client directory where the command is executed can directly access this relative path. Flink can transfer the keystore and truststore files using either of the following methods:
      • Add the -t option to the CLI yarn-session.sh command of Flink to transfer the keystore and truststore files to execution nodes. The following is an example.
        ./bin/yarn-session.sh -t ssl/ -n 2
      • Add the -yt option to the flink run command to transfer the keystore and truststore files to execution nodes. The following is an example.
        ./bin/flink run -yt ssl/ -ys 3 -yn 3 -m yarn-cluster -c org.apache.flink.examples.java.wordcount.WordCount /opt/client/Flink/flink/examples/batch/WordCount.jar
        • In the preceding example, ssl/ is the sub-directory of the Flink client directory and is used to store configuration files of the SSL keystore and truststore.
        • The relative path of ssl/ must be accessible from the current path where the Flink client command is executed.
    • If the keystore or truststore file path is an absolute path, the keystore and truststore files must exist in the absolute path on Flink Client and all nodes. In addition, the user who submits the job must have permission to read the files.

      Either of the following methods can be used to run applications. The -t or -yt option does not need to be added to transfer the keystore and truststore files.

      • Run the CLI yarn-session.sh command of Flink to execute applications. The following is an example.
        ./bin/yarn-session.sh -n 2
      • Run the flink run command to execute applications. The following is an example.
        ./bin/flink run  -ys 3 -yn 3 -m yarn-cluster -c org.apache.flink.examples.java.wordcount.WordCount /opt/client/Flink/flink/examples/batch/WordCount.jar

ACL

In HA mode of Flink, ZooKeeper can be used to manage clusters and discover services. Zookeeper supports SASL ACL. Only users who have passed the SASL (Kerberos) authentication have permission to operate files on ZooKeeper. To enable SASL ACL, configure the following parameters in the Flink configuration file.

high-availability.zookeeper.client.acl: creator
zookeeper.sasl.disable: false

For details about the configuration items, see Table 10.

Web security

Encoding Specifications

Note: The same encoding mode is used on the web service client and server to prevent garbled characters and to implement input verification.

Security hardening: Response messages of web servers are encoded using UTF-8.

Whitelist-based Filtering of IP Addresses

Note: To prevent unauthorized users from logging in to the web servers, add an IP filter on the web servers to filter out invalid requests of source IP addresses.

Security: Add jobmanager.web.allow-access-address to enable the IP filter. By default, only YARN users are supported.

Flink does not involve security risks of client storage, WebWorker, WebRTC, and WebSocket.

Preventing Absolute Paths of Files Being Sent to the Client

Note: If an absolute path is sent to a client, the directory structure of the server is exposed, helping attackers traverse and attack the system.

Security hardening: If the Flink configuration file contains a parameter starting with a slash (/), delete the first-level directory.

Security Statement

  • All security functions of Flink are provided by the open source community or self-developed. Security features, such as authentication and SSL encrypted transmission, that need to be configured by users, may affect performance.
  • As a big data computing and analysis platform, Flink does not detect sensitive information. Therefore, users need to ensure that the input data is anonymized.
  • Users need to evaluate whether configurations are secure as required.
  • For any security-related problems, contact customer service.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel