On this page

Brief Introduction to API for Kafka
Protocol Description for Accessing Kafka
ACL Settings for Topic
Use of New and Old Kafka APIs in Different Scenarios

Show all

Help Center/ MapReduce Service/ Troubleshooting/ Using Kafka/ Safety Instructions on Using Kafka

Safety Instructions on Using Kafka

Updated on 2024-09-18 GMT+08:00

View PDF

Brief Introduction to API for Kafka

New Producer API
Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.
Old Producer API
Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.
New Consumer API
Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.
Old Consumer API
Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

NOTE:

New Producer API and new Consumer API are called new API in general in the document.

Protocol Description for Accessing Kafka

The protocols used to access Kafka are as follows: PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL.

When Kafka is started, access monitoring using the PLAINTEXT and SASL_PLAINTEXT protocols is started. You can set ssl.mode.enable to true in Kafka service configurations to start monitoring using SSL and SASL_SSL protocols.

The following table describes the four protocols:

Protocol Type	Description	Supported API	Default Port
PLAINTEXT	Supports plaintext access without authentication.	New and old APIs	9092
SASL_PLAINTEXT	Supports plaintext access with Kerberos authentication.	New API	21007
SSL	Supports SSL-encrypted access without authentication.	New API	9093
SASL_SSL	Supports SSL-encrypted access with Kerberos authentication.	New API	21009

ACL Settings for Topic

Kafka supports secure access. Therefore, users can set the ACL for topics to control that different users access different topics. To view and set the permission information about a topic, run the kafka-acls.sh script on the Linux client.

Scenarios

Assign Kafka users with specific permissions for related topics based on service requirements.

The following table describes default Kafka user groups.

User Group	Description
kafkaadmin	Kafka administrator group. Users added to this group have the permissions to create, delete, authorize, as well as read from and write data to all topics.
kafkasuperuser	Users added to this group have permissions to read data from and write data to all topics.
kafka	Kafka common user group. If users in this group want to read data from and write data to a specific topic, the users in the kafkaadmin group must grant permissions to users in this group.

Prerequisites
1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
2. The Kafka client has been installed.
Procedure
1. Log in to the node where the Kafka client is installed as the client installation user.
2. Switch to the Kafka client installation directory, for example, /opt/kafkaclient.
  cd /opt/kafkaclient
3. Run the following command to configure environment variables:
  source bigdata_env
4. Run the following command to perform user authentication (skip this step for a cluster in common mode):
  kinit Component service user
5. Run the following command to switch to the Kafka client installation directory:
  cd Kafka/kafka/bin
6. The following describes the commands commonly used for user authorization when kafka-acl.sh is used:
  - View the permission control list of a topic:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --list --topic <Topic name>
  - Add the Producer permission for a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --producer --topic <Topic name>
  - Remove the Producer permission from a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --producer --topic <Topic name>
  - Add the Consumer permission for a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>
  - Remove the Consumer permission from a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>
    
    NOTE:
    
    In MRS 1.6.3 or earlier, the default ZooKeeper port number is 24002 regardless of whether Kerberos authentication is enabled for the cluster. In MRS 1.6.3 or later, the default ZooKeeper port number is 2181 regardless of whether Kerberos authentication is enabled for the cluster.

Use of New and Old Kafka APIs in Different Scenarios

Scenario 1: accessing the topic with an ACL

Used API	User Group	Client Parameter	Server Parameter	Access Port
New API	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group In the kafka group and be authorized	security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka	-	sasl.port (The default number is 21007.)
security.protocol=SASL_SSL sasl.kerberos.service.name = kafka	Set ssl.mode.enable to true.	sasl-ssl.port (The default port number is 21009.)
Old API	N/A	N/A	N/A	N/A

Used API

User Group

Client Parameter

Server Parameter

Access Port

New API

Users need to meet one of the following conditions:

In the administrator group
In the kafkaadmin group
In the kafkasuperuser group
In the kafka group and be authorized

security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka

sasl.port (The default number is 21007.)

security.protocol=SASL_SSL sasl.kerberos.service.name = kafka

Set ssl.mode.enable to true.

sasl-ssl.port (The default port number is 21009.)

Old API

N/A

Scenario 2: accessing the topic without an ACL

Used API	User Group	Client Parameter	Server Parameter	Access Port
New API	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group	security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka	-	sasl.port (The default number is 21007.)
	Users are in the kafka group.		Set allow.everyone.if.no.acl.found to true.	sasl.port (The default number is 21007.)
	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group	security.protocol=SASL_SSLsasl.kerberos.service.name = kafka	Set ssl-enable to true.	sasl-ssl.port (The default port number is 21009.)
	Users are in the kafka group.		Set allow.everyone.if.no.acl.found to true. Set ssl-enable to true.	sasl-ssl.port (The default port number is 21009.)
	-	security.protocol=PLAINTEXT	Set allow.everyone.if.no.acl.found to true.	port (The default number is 21005.)
	-	security.protocol=SSL	Set allow.everyone.if.no.acl.found to true. Set ssl-enable to true.	ssl.port (The default number is 21008.)
Old Producer	-	-	Set allow.everyone.if.no.acl.found to true.	port (The default number is 21005.)
Old Consumer	-	-	Set allow.everyone.if.no.acl.found to true.	ZooKeeper service port: clientPort (The default number is 24002.)

Parent topic: Using Kafka

Previous topic: Leader for the Created Kafka Topic Partition Is Displayed as none

Next topic: Obtaining Kafka Consumer Offset Information

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot