Help Center > > User Guide> MRS Cluster Component Operation Guide> Using Flink> Security Configuration

Security Configuration

Updated at:Sep 08, 2020 GMT+08:00

Flink provides the following security features:

  • All Flink cluster components support authentication.

    Kerberos authentication is supported between internal and external components of the Flink cluster.

  • Flink cluster components support SSL encrypted transmission.
  • SSL encrypted transmission is supported between internal components of the Flink cluster, for example, between the Flink client and JobManager, JobManager and TaskManager, and TaskManagers.
  • Flink web security hardening
    • Whitelist filtering. The Flink web can only be accessed through the Yarn proxy.
    • Security header enhancement.
  • In Flink clusters, listening ports of components can be configured.
  • In HA mode, ACL is supported.

Interconnecting with Kafka

Flink sample project data is stored in Kafka. A user with Kafka permission can send data to Kafka and receive data from it.

  1. Ensure that a cluster containing HDFS, Yarn, Flink, and Kafka has been successfully installed.
  2. Create a topic.

    • Run a Linux command line to create a topic. Before running a command, run the kinit command, for example, kinit flinkuser, to authenticate the human-machine account.

      flinkuser is created by yourself and has permission to create Kafka topics. For details, see Preparing a Development User.

      The following provides the format of the command for creating a topic. {zkQuorum} indicates ZooKeeper cluster information in IP:port format. {Topic} indicates the topic name.

      bin/kafka-topics.sh --create --zookeeper {zkQuorum}/kafka --replication-factor 1 --partitions 5 --topic {Topic}

      Assume the topic name is topic 1. The command for creating this topic is displayed as follows:

      /opt/client/Kafka/kafka/bin/kafka-topics.sh --create --zookeeper 10.96.101.32:2181,10.96.101.251:24002,10.96.101.177:24002,10.91.8.160:24002/kafka --replication-factor 1 --partitions 5 --topic topic1
    • Configuring the permission of topics on the server

      Go to the cluster details page. Choose Components > Kafka > Service Configuration, change Basic to All in the parameter type drop-down box, and change the value of the Broker parameter allow.everyone.if.no.acl.found of Kafka to true.

  3. Perform security authentication.

    • Kerberos authentication configuration
      • Client configuration

        In the Flink configuration file flink-conf.yaml, add configurations about Kerberos authentication. For example, add KafkaClient in contexts as follows:

        security.kerberos.login.keytab: /home/demo/flink/release/keytab/flinkuser.keytab
        security.kerberos.login.principal: flinkuser
        security.kerberos.login.contexts: Client,KafkaClient
        security.kerberos.login.use-ticket-cache: false
      • Running parameters

        Running parameters about the SASL_PLAINTEXT protocol are displayed as follows:

        --topic topic1 --bootstrap.servers 10.96.101.32:21007 --security.protocol SASL_PLAINTEXT  --sasl.kerberos.service.name kafka //10.96.101.32:21007 indicates the IP:port of the Kafka server.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel