Reading the Content of the __consumer_offsets Internal Topic

Issue

How does Kafka save the offset of a Consumer to the __consumer_offsets of internal topics?

Procedure

1. Log in to the node where the Kafka client is installed as the client installation user.
2. Switch to the Kafka client installation directory, for example, /opt/client.

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Run the following command to perform user authentication (skip this step for a cluster in common mode):

   kinit Component service user

5. Run the following command to switch to the Kafka client installation directory:

   cd Kafka/kafka/bin

6. Run the following command to obtain Consumer offset metric information:

   kafka-console-consumer.sh --topic __consumer_offsets --zookeeper <zk_host:port>/kafka --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config <property file> --from-beginning

   Add the following content to the <property file> configuration file:

   exclude.internal.topics = false

   Example:

   kafka-console-consumer.sh --topic __consumer_offsets --zookeeper 10.5.144.2:2181/kafka --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config ../config/consumer.properties --from-beginning