Leader for the Created Kafka Topic Partition Is Displayed as none

Symptom

When a user creates a topic using the Kafka client command, the leader for the created topic partition is displayed as none.

[root@10-10-144-2 client]# 
kafka-topics.sh --create  --replication-factor 1 --partitions 2 --topic test --zookeeper 10.6.92.36:2181/kafka

Created topic "test".

[root@10-10-144-2 client]# 
kafka-topics.sh --describe --zookeeper 10.6.92.36:2181/kafka

Topic:test      PartitionCount:2        ReplicationFactor:2     Configs:
        Topic: test     Partition: 0    Leader: none       Replicas: 2,3   Isr:
        Topic: test     Partition: 1    Leader: none       Replicas: 3,1   Isr:

Possible Causes

• The Kafka service is not running.
• The user group information cannot be found.

Cause Analysis

1. Check the Kafka service status and monitoring metrics.
   • MRS Manager: Log in to MRS Manager and choose Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
   • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
2. Obtain the Controller node information on the Kafka overview page.
3. Log in to the node where the Controller resides, and run the cd /var/log/Bigdata/kafka/broker command to go to the node log directory. The state-change.log contains "NoAuthException", which indicates that the ZooKeeper permission is incorrect.

   2018-05-31 09:20:42,436 | ERROR | [ZkClient-EventThread-34-10.6.92.36:24002,10.6.92.37:24002,10.6.92.38:24002/kafka] | Controller 4 epoch 6 initiated state change for partition [test,1] from NewPartition to OnlinePartition failed | state.change.logger (Logging.scala:103) 
   
    org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /brokers/topics/test/partitions
    at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1000)
    at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:527)
    at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:293)

4. Check on ZooKeeper audit logs recorded in the specified period also indicates that the permission is abnormal.

   2018-05-31 09:20:42,421 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions/0/state	result=failure 
   2018-05-31 09:20:42,423 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions/0	result=failure 
   2018-05-31 09:20:42,435 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions	result=failure 
   2018-05-31 09:20:42,439 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions/1/state	result=failure 
   2018-05-31 09:20:42,441 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions/1	result=failure 
   2018-05-31 09:20:42,453 | ERROR | CommitProcWorkThread-1 | session=0xc3000007015d5a18	user=10.6.92.39,kafka/hadoop.hadoop.com@HADOOP.COM,kafka/hadoop.hadoop.com@HADOOP.COM	ip=10.6.92.39	operation=create znode	target=ZooKeeperServer	znode=/kafka/brokers/topics/test/partitions	result=failure 

5. Run the id -Gn kafka command on each ZooKeeper instance node. It is found that user group information cannot be queried on a node.

   [root @bdpsit3ap03 ~]# id -Gn kafka
   id:  kafka: No such user
   [root @bdpsit3ap03 ~]#

6. In an MRS cluster, user management is provided by the LDAP service and depends on the SSSD (Red Hat) and NSCD (SUSE) services of OSs. The process from creating a user to synchronizing the user to the SSSD service takes some time. If the user does not take effect or the SSSD version has bugs, the user may be invalid on the ZooKeeper node in some cases, which causes topic creation failures.

Solution

1. Restart the SSD/NSCD service.
   • RedHat

     service sssd restart

   • SUSE

     sevice nscd restart

2. After restarting related services, run the id username command on the active ResourceManager node to check whether the user information is valid.