Help Center/ MapReduce Service/ Troubleshooting/ Using Kafka/ Safety Instructions on Using Kafka
Updated on 2024-09-18 GMT+08:00
View PDF
Share

Safety Instructions on Using Kafka

Brief Introduction to API for Kafka

  • New Producer API

    Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.

  • Old Producer API

    Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.

  • New Consumer API

    Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.

  • Old Consumer API

    Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

New Producer API and new Consumer API are called new API in general in the document.

Protocol Description for Accessing Kafka

The protocols used to access Kafka are as follows: PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL.

When Kafka is started, access monitoring using the PLAINTEXT and SASL_PLAINTEXT protocols is started. You can set ssl.mode.enable to true in Kafka service configurations to start monitoring using SSL and SASL_SSL protocols.

The following table describes the four protocols:

Protocol Type

Description

Supported API

Default Port

PLAINTEXT

Supports plaintext access without authentication.

New and old APIs

9092

SASL_PLAINTEXT

Supports plaintext access with Kerberos authentication.

New API

21007

SSL

Supports SSL-encrypted access without authentication.

New API

9093

SASL_SSL

Supports SSL-encrypted access with Kerberos authentication.

New API

21009

ACL Settings for Topic

Kafka supports secure access. Therefore, users can set the ACL for topics to control that different users access different topics. To view and set the permission information about a topic, run the kafka-acls.sh script on the Linux client.

  • Scenarios

    Assign Kafka users with specific permissions for related topics based on service requirements.

    The following table describes default Kafka user groups.

    User Group

    Description

    kafkaadmin

    Kafka administrator group. Users added to this group have the permissions to create, delete, authorize, as well as read from and write data to all topics.

    kafkasuperuser

    Users added to this group have permissions to read data from and write data to all topics.

    kafka

    Kafka common user group. If users in this group want to read data from and write data to a specific topic, the users in the kafkaadmin group must grant permissions to users in this group.
  • Prerequisites
    1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
    2. The Kafka client has been installed.
  • Procedure
    1. Log in to the node where the Kafka client is installed as the client installation user.
    2. Switch to the Kafka client installation directory, for example, /opt/kafkaclient.

      cd /opt/kafkaclient

    3. Run the following command to configure environment variables:

      source bigdata_env

    4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

      kinit Component service user

    5. Run the following command to switch to the Kafka client installation directory:

      cd Kafka/kafka/bin

    6. The following describes the commands commonly used for user authorization when kafka-acl.sh is used:
      • View the permission control list of a topic:

        ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --list --topic <Topic name>

      • Add the Producer permission for a user:

        ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --producer --topic <Topic name>

      • Remove the Producer permission from a user:

        ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --producer --topic <Topic name>

      • Add the Consumer permission for a user:

        ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>

      • Remove the Consumer permission from a user:

        ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>

        In MRS 1.6.3 or earlier, the default ZooKeeper port number is 24002 regardless of whether Kerberos authentication is enabled for the cluster. In MRS 1.6.3 or later, the default ZooKeeper port number is 2181 regardless of whether Kerberos authentication is enabled for the cluster.

Use of New and Old Kafka APIs in Different Scenarios

  • Scenario 1: accessing the topic with an ACL

    Used API

    User Group

    Client Parameter

    Server Parameter

    Access Port

    New API

    Users need to meet one of the following conditions:

    • In the administrator group
    • In the kafkaadmin group
    • In the kafkasuperuser group
    • In the kafka group and be authorized

    security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka

    -

    sasl.port (The default number is 21007.)

    security.protocol=SASL_SSL sasl.kerberos.service.name = kafka

    Set ssl.mode.enable to true.

    sasl-ssl.port (The default port number is 21009.)

    Old API

    N/A

    N/A

    N/A

    N/A
  • Scenario 2: accessing the topic without an ACL

    Used API

    User Group

    Client Parameter

    Server Parameter

    Access Port

    New API

    Users need to meet one of the following conditions:

    • In the administrator group
    • In the kafkaadmin group
    • In the kafkasuperuser group

    security.protocol=SASL_PLAINTEXT

    sasl.kerberos.service.name = kafka

    -

    sasl.port (The default number is 21007.)

    Users are in the kafka group.

    Set allow.everyone.if.no.acl.found to true.

    sasl.port (The default number is 21007.)

    Users need to meet one of the following conditions:

    • In the administrator group
    • In the kafkaadmin group
    • In the kafkasuperuser group

    security.protocol=SASL_SSLsasl.kerberos.service.name = kafka

    Set ssl-enable to true.

    sasl-ssl.port (The default port number is 21009.)

    Users are in the kafka group.

    Set allow.everyone.if.no.acl.found to true.

    Set ssl-enable to true.

    sasl-ssl.port (The default port number is 21009.)

    -

    security.protocol=PLAINTEXT

    Set allow.everyone.if.no.acl.found to true.

    port (The default number is 21005.)

    -

    security.protocol=SSL

    Set allow.everyone.if.no.acl.found to true.

    Set ssl-enable to true.

    ssl.port (The default number is 21008.)

    Old Producer

    -

    -

    Set allow.everyone.if.no.acl.found to true.

    port (The default number is 21005.)

    Old Consumer

    -

    -

    Set allow.everyone.if.no.acl.found to true.

    ZooKeeper service port: clientPort (The default number is 24002.)
Parent topic: Using Kafka