Obtaining Kafka Consumer Offset Information

Symptom

How do I obtain Kafka Consumer offset information when using Kafka Consumer to consume data?

Kafka APIs

• New Producer API

  Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.

• Old Producer API

  Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.

• New Consumer API

  Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.

• Old Consumer API

  Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

New Producer API and new Consumer API are called new API in general in the document.

Procedure

Old Consumer API

• Prerequisites
  1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
  2. The Kafka client has been installed.
• Procedure
  1. Log in to the node where the Kafka client is installed as the client installation user.
  2. Switch to the Kafka client installation directory, for example, /opt/kafkaclient.

     cd /opt/kafkaclient

  3. Run the following command to configure environment variables:

     source bigdata_env

  4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

     kinit Component service user

  5. Run the following command to switch to the Kafka client installation directory:

     cd Kafka/kafka/bin

  6. Run the following command to obtain Consumer offset metric information:

      bin/kafka-consumer-groups.sh --zookeeper <zookeeper_host:port>/kafka --list 

     bin/kafka-consumer-groups.sh --zookeeper <zookeeper_host:port>/kafka --describe --group test-consumer-group

     Example:

     kafka-consumer-groups.sh --zookeeper 192.168.100.100:2181/kafka --list 
     kafka-consumer-groups.sh --zookeeper 192.168.100.100:2181/kafka --describe --group test-consumer-group

New Consumer API

• Prerequisites
  1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).
  2. The Kafka client has been installed.
• Procedure
  1. Log in to the node where the Kafka client is installed as the client installation user.
  2. Switch to the Kafka client installation directory, for example, /opt/client.

     cd /opt/client

  3. Run the following command to configure environment variables:

     source bigdata_env

  4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

     kinit Component service user

  5. Run the following command to switch to the Kafka client installation directory:

     cd Kafka/kafka/bin

  6. Run the following command to obtain Consumer offset metric information:

     kafka-consumer-groups.sh --bootstrap-server <broker_host:port> --describe --group my-group

     Example:

     kafka-consumer-groups.sh --bootstrap-server 192.168.100.100:9092 --describe --group my-group