Safety Instructions on Using Kafka

Brief Introduction to API for Kafka

New Producer API
Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.
Old Producer API
Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.
New Consumer API
Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.
Old Consumer API
Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

New Producer API and new Consumer API are called new API in general in the document.

Protocol Description for Accessing Kafka

The protocols used to access Kafka are as follows: PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL.

When Kafka service is started, the listeners using the PLAINTEXT and SASL_PLAINTEXT protocols are started. You can set ssl.mode.enable to true in Kafka service configuration to start listeners using SSL and SASL_SSL protocols.

The following table describes the four protocols:

Protocol Type	Description	Supported API	Default Port
PLAINTEXT	Supports plaintext access without authentication.	New and old APIs	9092
SASL_PLAINTEXT	Supports plaintext access with Kerberos authentication.	New API	21007
SSL	Supports SSL-encrypted access without authentication.	New API	9093
SASL_SSL	Supports SSL-encrypted access with Kerberos authentication.	New API	21009

ACL Settings for Topic

Kafka supports secure access. Therefore, users can set the ACL for topics to control that different users access different topics. To view and set the permission information about a topic, run the kafka-acls.sh script on the Linux client.

Scenarios

Assign Kafka users with specific permissions for related topics based on service requirements.

The following table describes default Kafka user groups.

User Group	Description
kafkaadmin	Kafka administrator group. Users added to this group have the permissions to create, delete, authorize, as well as read from and write data to all topics.
kafkasuperuser	Users added to this group have permissions to read data from and write data to all topics.
kafka	Kafka common user group. If users in this group want to read data from and write data to a specific topic, the users in the kafkaadmin group must grant permissions to users in this group.

Prerequisites
1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
2. The Kafka client has been installed.
Procedure
1. Log in to the node where the Kafka client is installed as the client installation user.
2. Switch to the Kafka client installation directory, for example, /opt/kafkaclient.
  cd /opt/kafkaclient
3. Run the following command to configure environment variables:
  source bigdata_env
4. Run the following command to perform user authentication (skip this step for a cluster in common mode):
  kinit Component service user
5. Run the following command to switch to the Kafka client installation directory:
  cd Kafka/kafka/bin
6. The following describes the commands commonly used for user authorization when kafka-acl.sh is used:
  - View the permission control list of a topic:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --list --topic <Topic name>
  - Add the Producer permission for a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --producer --topic <Topic name>
  - Remove the Producer permission from a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --producer --topic <Topic name>
  - Add the Consumer permission for a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --add --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>
  - Remove the Consumer permission from a user:
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=<ZooKeeper cluster service IP:2181/kafka > --remove --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>

Use of New and Old Kafka APIs in Different Scenarios

Scenario 1: accessing the topic with an ACL

Used API	User Group	Client Parameter	Server Parameter	Access Port
New API	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group In the kafka group and be authorized	security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka	-	sasl.port (The default number is 21007.)
security.protocol=SASL_SSL sasl.kerberos.service.name = kafka	Set ssl.mode.enable to true.	sasl-ssl.port (The default port number is 21009.)
Old API	N/A	N/A	N/A	N/A

Used API

User Group

Client Parameter

Server Parameter

Access Port

New API

Users need to meet one of the following conditions:

In the administrator group
In the kafkaadmin group
In the kafkasuperuser group
In the kafka group and be authorized

security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka

sasl.port (The default number is 21007.)

security.protocol=SASL_SSL sasl.kerberos.service.name = kafka

Set ssl.mode.enable to true.

sasl-ssl.port (The default port number is 21009.)

Old API

N/A

Scenario 2: accessing the topic without an ACL

Used API	User Group	Client Parameter	Server Parameter	Access Port
New API	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group	security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name = kafka	-	sasl.port (The default number is 21007.)
	Users are in the kafka group.		Set allow.everyone.if.no.acl.found to true.	sasl.port (The default number is 21007.)
	Users need to meet one of the following conditions: In the administrator group In the kafkaadmin group In the kafkasuperuser group	security.protocol=SASL_SSLsasl.kerberos.service.name = kafka	Set ssl-enable to true.	sasl-ssl.port (The default port number is 21009.)
	Users are in the kafka group.		Set allow.everyone.if.no.acl.found to true. Set ssl-enable to true.	sasl-ssl.port (The default port number is 21009.)
	-	security.protocol=PLAINTEXT	Set allow.everyone.if.no.acl.found to true.	port (The default number is 21005.)
	-	security.protocol=SSL	Set allow.everyone.if.no.acl.found to true. Set ssl-enable to true.	ssl.port (The default number is 21008.)
Old Producer	-	-	Set allow.everyone.if.no.acl.found to true.	port (The default number is 21005.)
Old Consumer	-	-	Set allow.everyone.if.no.acl.found to true.	ZooKeeper service port: clientPort (The default number is 24002.)