Help Center/ MapReduce Service/ Troubleshooting/ Using Kafka/ Obtaining Kafka Consumer Offset Information
Updated on 2022-11-04 GMT+08:00

Obtaining Kafka Consumer Offset Information

Symptom

How do I obtain Kafka Consumer offset information when using Kafka Consumer to consume data?

Kafka APIs

  • New Producer API

    Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.

  • Old Producer API

    Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.

  • New Consumer API

    Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.

  • Old Consumer API

    Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

New Producer API and new Consumer API are called new API in general in the document.

Procedure

Old Consumer API

  • Prerequisites
    1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
    2. The Kafka client has been installed.
  • Procedure
    1. Log in to the node where the Kafka client is installed as the client installation user.
    2. Switch to the Kafka client installation directory, for example, /opt/kafkaclient.

      cd /opt/kafkaclient

    3. Run the following command to configure environment variables:

      source bigdata_env

    4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

      kinit Component service user

    5. Run the following command to switch to the Kafka client installation directory:

      cd Kafka/kafka/bin

    6. Run the following command to obtain Consumer offset metric information:

       bin/kafka-consumer-groups.sh --zookeeper <zookeeper_host:port>/kafka --list 

      bin/kafka-consumer-groups.sh --zookeeper <zookeeper_host:port>/kafka --describe --group test-consumer-group

      Example:

      kafka-consumer-groups.sh --zookeeper 192.168.100.100:2181/kafka --list 
      kafka-consumer-groups.sh --zookeeper 192.168.100.100:2181/kafka --describe --group test-consumer-group

New Consumer API

  • Prerequisites
    1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
    2. The Kafka client has been installed.
  • Procedure
    1. Log in to the node where the Kafka client is installed as the client installation user.
    2. Switch to the Kafka client installation directory, for example, /opt/client.

      cd /opt/client

    3. Run the following command to configure environment variables:

      source bigdata_env

    4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

      kinit Component service user

    5. Run the following command to switch to the Kafka client installation directory:

      cd Kafka/kafka/bin

    6. Run the following command to obtain Consumer offset metric information:

      kafka-consumer-groups.sh --bootstrap-server <broker_host:port> --describe --group my-group

      Example:

      kafka-consumer-groups.sh --bootstrap-server 192.168.100.100:9092 --describe --group my-group