Obtaining Topic Distribution Information

Issue

How do I obtain topic distribution information in a Broker instance?

Preparations

• Prerequisites

  The Kafka and ZooKeeper clients have been installed.

• Procedure
  1. Log in to the node where the Kafka client is installed as the client installation user.
  2. Switch to the Kafka client installation directory, for example, /opt/client.

     cd /opt/client

  3. Run the following command to configure environment variables:

     source bigdata_env

  4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

     kinit Component service user

  5. Run the following command to switch to the Kafka client installation directory:

     cd Kafka/kafka/bin

  6. Run the Kafka commands to obtain the topic assignment information and copy synchronization information, and check the return result.

     kafka-topics.sh --describe --zookeeper <zk_host:port/chroot>

     Example:

     [root@mgtdat-sh-3-01-3 client]#kafka-topics.sh --describe --zookeeper 10.149.0.90:2181/kafka
     Topic:topic1   PartitionCount:2  ReplicationFactor:2     Configs:
     Topic: topic1  Partition: 0 Leader: 26 Replicas: 23,25 Isr: 26
     Topic: topic1  Partition: 1 Leader: 24 Replicas: 24,23 Isr: 24,23

     In the preceding information, Replicas indicates the replica assignment information and Isr indicates the replica synchronization information.

Solution 1

1. Query the Broker ID mapping in ZooKeeper.

   sh zkCli.sh -server <zk_host:port>

2. Run the following command on the ZooKeeper client:

   ls /kafka/brokers/ids

   get/kafka/brokers/ids/<queried Broker ID>

   Example:

   [root@node-master1gAMQ kafka]# zkCli.sh -server node-master1gAMQ:2181
   Connecting to node-master1gAMQ:2181
   Welcome to ZooKeeper!
   JLine support is enabled
   
   WATCHER::
   
   WatchedEvent state:SyncConnected type:None path:null
   [zk: node-master1gAMQ:2181(CONNECTED) 0] ls /kafka/brokers/
   ids      seqid    topics
   [zk: node-master1gAMQ:2181(CONNECTED) 0] ls /kafka/brokers/ids
   [1]
   [zk: node-master1gAMQ:2181(CONNECTED) 1] get /kafka/brokers/ids/1
   {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT","SSL":"SSL"},"endpoints":["PLAINTEXT://192.168.2.242:9092","SSL://192.168.2.242:9093"],"rack":"/default/rack0","jmx_port":21006,"host":"192.168.2.242","timestamp":"1580886124398","port":9092,"version":4}
   [zk: node-master1gAMQ:2181(CONNECTED) 2]
   

Solution 2

Obtain the mapping between nodes and Broker IDs.

kafka-broker-info.sh --zookeeper <zk_host:port/chroot>

Example:

[root@node-master1gAMQ kafka]# bin/kafka-broker-info.sh --zookeeper 192.168.2.70:2181/kafka
Broker_ID     IP_Address
--------------------------
  1           192.168.2.242