Producer Fails to Send Data and "TOPIC_AUTHORIZATION_FAILED" Is Thrown

Symptom

An MRS cluster is installed, and ZooKeeper and Kafka are installed in the cluster.

Possible Causes

1. The Kafka service is abnormal.
2. The Producer client adopts non-security access and access is disabled on the server.
3. The Producer client adopts non-security access and ACL is set for Kafka topics.

Cause Analysis

The possible reasons why Producer fails to send data to Kafka may be related to Producer or Kafka.

1. Check the Kafka service status:
   • MRS Manager: Log in to MRS Manager and choose Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
   • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka.

     Check the Kafka status. It is found that the status is good and the monitoring metrics are correctly displayed.

2. Check the Producer client logs. The logs contain the error information "TOPIC_AUTHORIZATION_FAILED."

   [root@10-10-144-2 client]# kafka-console-producer.sh --broker-list 10.5.144.2:9092 --topic test
   1
   [2017-01-24 16:58:36,671] WARN Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
   [2017-01-24 16:58:36,672] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: Not authorized to access topics: [test] (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

   Producer accesses Kafka using port 9092, which is a non-security port.

3. On Manager, check the current Kafka cluster configuration. It is found that the customized configuration allow.everyone.if.no.acl.found=false is not configured.
   • MRS Manager portal: Log in to MRS Manager and choose Services > Kafka > Configuration.
   • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka > Configuration.
4. If ACL is set to false, port 9092 cannot be used for access.
5. Check the Producer client logs. The logs contain the error information "TOPIC_AUTHORIZATION_FAILED."

   [root@10-10-144-2 client]# kafka-console-producer.sh --broker-list 10.5.144.2:21005 --topic test_acl
   1
   [2017-01-25 11:09:40,012] WARN Error while fetching metadata with correlation id 0 : {test_acl=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
   [2017-01-25 11:09:40,013] ERROR Error when sending message to topic test_acl with key: null, value: 1 bytes with error: Not authorized to access topics: [test_acl] (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
   [2017-01-25 11:14:40,010] WARN Error while fetching metadata with correlation id 1 : {test_acl=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)

   Producer accesses Kafka using port 21005, which is a non-security port.

6. Run the client command to check the ACL permission of the topic.

   [root@10-10-144-2 client]# kafka-acls.sh --authorizer-properties zookeeper.connect=10.5.144.2:24002/kafka --list --topic topic_acl
   Current ACLs for resource `Topic:topic_acl`: 
    User:test_user has Allow permission for operations: Describe from hosts: *
   User:test_user has Allow permission for operations: Write from hosts: * 

   If ACL is set for the topic, port 9092 cannot be used for access.

7. Check the Producer client logs. The logs contain the error information "TOPIC_AUTHORIZATION_FAILED."

   [root@10-10-144-2 client]# kafka-console-producer.sh --broker-list 10.5.144.2:21007 --topic topic_acl --producer.config /opt/client/Kafka/kafka/config/producer.properties
   1
   [2017-01-25 12:43:58,506] WARN Error while fetching metadata with correlation id 0 : {topic_acl=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
   [2017-01-25 12:43:58,507] ERROR Error when sending message to topic topic_acl with key: null, value: 1 bytes with error: Not authorized to access topics: [topic_acl] (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

   Producer uses port 21007 to access Kafka.

8. Run the client command klist to query the current authenticated user.

   [root@10-10-144-2 client]# klist
   Ticket cache: FILE:/tmp/krb5cc_0
   Default principal: test@HADOOP.COM
   
   Valid starting     Expires            Service principal
   01/25/17 11:06:48  01/26/17 11:06:45  krbtgt/HADOOP.COM@HADOOP.COM

   The test user is used in this example.

9. Run the client command to check the ACL permission of the topic.

   [root@10-10-144-2 client]# kafka-acls.sh --authorizer-properties zookeeper.connect=10.5.144.2:2181/kafka --list --topic topic_acl
   Current ACLs for resource `Topic:topic_acl`: 
    User:test_user has Allow permission for operations: Describe from hosts: *
   User:test_user has Allow permission for operations: Write from hosts: * 

   After ACL is set for the topic, user test_user has Producer permission. User test has no permission to perform Producer operations.

   For details about the solution, see 2.

10. Log in to Kafka Broker using SSH.

    Run the cd /var/log/Bigdata/kafka/broker command to go to the log directory.

    Check the kafka-authorizer.log file. It shows that the user does not belong to the kafka or kafkaadmin group.

    2017-01-25 13:26:33,648 | INFO  | [kafka-request-handler-0] | The principal is test, belongs to Group : [hadoop, ficommon] | kafka.authorizer.logger (SimpleAclAuthorizer.scala:169)
    2017-01-25 13:26:33,648 | WARN  | [kafka-request-handler-0] | The user is not belongs to kafka or kafkaadmin group, authorize failed! | kafka.authorizer.logger (SimpleAclAuthorizer.scala:170)

    For details about the solution, see 3.

Solution

1. Set allow.everyone.if.no.acl.found to true and restart the Kafka service.
2. Use the account with permission for login.

   Example:

   kinit test_user

   Alternatively, grant the user with related permission.

   

   This operation must be performed by the Kafka administrator (belonging to the kafkaadmin group).

   Example:

   kafka-acls.sh --authorizer-properties zookeeper.connect=10.5.144.2:2181/kafka --topic topic_acl --producer --add --allow-principal User:test

   [root@10-10-144-2 client]# kafka-acls.sh --authorizer-properties zookeeper.connect=8.5.144.2:2181/kafka --list --topic topic_acl
   Current ACLs for resource `Topic:topic_acl`: 
    User:test_user has Allow permission for operations: Describe from hosts: *
    User:test_user has Allow permission for operations: Write from hosts: *
     User:test has Allow permission for operations: Describe from hosts: *
     User:test has Allow permission for operations: Write from hosts: *

3. Add the user to the kafka or kafkaadmin group.