Checking the Consumption Status of Consumer Group

Scenario

View the current consumption information on the client as an MRS cluster administrator based on service requirements.

Prerequisites

• The MRS cluster administrator has understood service requirements and prepared a system user.

Procedure

1. Log in as a client installation user to the node on which the Kafka client is installed.
2. Switch to the Kafka client installation directory, for example, /opt/client.

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Run the following command to perform user authentication (skip this step in normal mode):

   kinit Component service user

5. Run the following command to switch to the Kafka client installation directory:

   cd Kafka/kafka/bin

6. Run the kafka-consumer-groups.sh command to check the current consumption status.
   • Check the Consumer Group list on Kafka saved by Offset:

     ./kafka-consumer-groups.sh --list --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --command-config ../config/consumer.properties

     Example: ./kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:21007 --list --command-config ../config/consumer.properties

   • Check the consumption status of Consumer Group on Kafka saved by Offset:

     ./kafka-consumer-groups.sh --describe --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --group Consumer group name --command-config ../config/consumer.properties

     Example: ./kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.1:21007 --group example-group --command-config ../config/consumer.properties

   • Query the statuses of multiple consumer groups in batches.

     ./kafka-consumer-groups.sh --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --list --state --command-config ../config/consumer.properties

     Example:

     • List the statuses of all consumer groups.

       ./kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.1:21007 --list --state --command-config ../config/consumer.properties

     • List all consumer groups in the stable state.

       ./kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.1:21007 --list --state stable --command-config ../config/consumer.properties

   • Print the offset and header of a topic.

     ./kafka-console-consumer.sh --bootstrap-server <Service IP address of any broker node:Port number of the Kafka cluster> --topic Topic name --from-beginning --property print.partition=true --property print.key=true --property print.timestamp=true --property print.offset=true --property print.headers=true --property key.separator='|' --consumer.config ../config/consumer.properties

     Example: ./kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:21007 --topic test --from-beginning --property print.partition=true --property print.key=true --property print.timestamp=true --property print.offset=true --property print.headers=true --property key.separator='|' --consumer.config ../config/consumer.properties

   

   1. Ensure that the current consumer is online and consumes data.
   2. Configure the group.id in the consumer.properties configuration file and --group in the command to the group to be queried.
   3. The Kafka cluster's IP port number is 21007 in security mode and 9092 in normal mode.