Help Center/ MapReduce Service/ Troubleshooting/ Using Kafka/ Obtaining Topic Distribution Information
Updated on 2023-11-30 GMT+08:00

Obtaining Topic Distribution Information

Issue

How do I obtain topic distribution information in a Broker instance?

Preparations

  • Prerequisites

    The Kafka and ZooKeeper clients have been installed.

  • Procedure
    1. Log in to the node where the Kafka client is installed as the client installation user.
    2. Switch to the Kafka client installation directory, for example, /opt/client.

      cd /opt/client

    3. Run the following command to configure environment variables:

      source bigdata_env

    4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

      kinit Component service user

    5. Run the following command to switch to the Kafka client installation directory:

      cd Kafka/kafka/bin

    6. Run the Kafka commands to obtain the topic assignment information and copy synchronization information, and check the return result.

      kafka-topics.sh --describe --zookeeper <zk_host:port/chroot>

      Example:

      [root@mgtdat-sh-3-01-3 client]#kafka-topics.sh --describe --zookeeper 10.149.0.90:2181/kafka
      Topic:topic1   PartitionCount:2  ReplicationFactor:2     Configs:
      Topic: topic1  Partition: 0 Leader: 26 Replicas: 23,25 Isr: 26
      Topic: topic1  Partition: 1 Leader: 24 Replicas: 24,23 Isr: 24,23

      In the preceding information, Replicas indicates the replica assignment information and Isr indicates the replica synchronization information.

Solution 1

  1. Query the Broker ID mapping in ZooKeeper.

    sh zkCli.sh -server <zk_host:port>

  2. Run the following command on the ZooKeeper client:

    ls /kafka/brokers/ids

    get/kafka/brokers/ids/<queried Broker ID>

    Example:

    [root@node-master1gAMQ kafka]# zkCli.sh -server node-master1gAMQ:2181
    Connecting to node-master1gAMQ:2181
    Welcome to ZooKeeper!
    JLine support is enabled
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    [zk: node-master1gAMQ:2181(CONNECTED) 0] ls /kafka/brokers/
    ids      seqid    topics
    [zk: node-master1gAMQ:2181(CONNECTED) 0] ls /kafka/brokers/ids
    [1]
    [zk: node-master1gAMQ:2181(CONNECTED) 1] get /kafka/brokers/ids/1
    {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT","SSL":"SSL"},"endpoints":["PLAINTEXT://192.168.2.242:9092","SSL://192.168.2.242:9093"],"rack":"/default/rack0","jmx_port":21006,"host":"192.168.2.242","timestamp":"1580886124398","port":9092,"version":4}
    [zk: node-master1gAMQ:2181(CONNECTED) 2]
    

Solution 2

Obtain the mapping between nodes and Broker IDs.

kafka-broker-info.sh --zookeeper <zk_host:port/chroot>

Example:

[root@node-master1gAMQ kafka]# bin/kafka-broker-info.sh --zookeeper 192.168.2.70:2181/kafka
Broker_ID     IP_Address
--------------------------
  1           192.168.2.242