Checking the Consumption Information of Kafka Topics

Updated on 2024-10-08 GMT+08:00

View PDF

Scenario

You can use the Kafka client or Kafka UI to view the current consumption information based on service requirements.

This section applies to MRS 3.x or later.

Prerequisites

When using the Kafka client, ensure that the following conditions are met:

The MRS cluster administrator has understood service requirements and prepared a system user.

The Kafka client has been installed.

Using the Kafka Client to View the Consumption Information

Log in as a client installation user to the node on which the Kafka client is installed.
Switch to the Kafka client installation directory, for example, /opt/client.

cd /opt/client
Run the following command to configure environment variables:

source bigdata_env
Run the following command to perform user authentication (skip this step in normal mode):

kinit Component service user
Run the following command to switch to the Kafka client installation directory:

cd Kafka/kafka/bin
Run the kafka-consumer-groups.sh command to check the current consumption status.
- Check the Consumer Group list on Kafka saved by Offset:
  ./kafka-consumer-groups.sh --list --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --command-config ../config/consumer.properties
  
  Example: ./kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:21007 --list --command-config ../config/consumer.properties
- Check the consumption status of Consumer Group on Kafka saved by Offset:
  ./kafka-consumer-groups.sh --describe --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --group Consumer group name --command-config ../config/consumer.properties
  
  Example: ./kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.1:21007 --group example-group --command-config ../config/consumer.properties
NOTICE:
1. Ensure that the current consumer is online and consumes data.
2. Configure the group.id in the consumer.properties configuration file and --group in the command to the group to be queried.
3. The Kafka cluster's IP port number is 21007 in security mode and 9092 in normal mode.

Using the Kafka UI to View the Consumption Information (for MRS 3.1.2 and Later)

Access the Kafka UI.
1. Log in to FusionInsight Manager as a user who has the permission to access the Kafka UI and choose Cluster > Services > Kafka.
  If you need to perform operations on the page, for example, creating a topic, you need to grant related permissions to users. For details, see Kafka User Permissions.
2. On the right of KafkaManager WebUI, click the URL to access the Kafka UI.
Click Consumers. On the consumer group details page that is displayed, you can view all consumer groups in the current cluster and the IP address of the node where each consumer group coordinator is located. In the upper right corner of the page, you can enter a consumer group name to search for the specified consumer group.
In the Consumer Summary area, you can view the existing consumer groups in the current cluster. You can click a consumer group name to view the topics consumed by the consumer group. Consumed topics can be in the pending or running state. pending indicates that the topic has been consumed but not being consumed. running indicates that the topic is being consumed. You can enter a topic name in the upper right corner of the dialog box to filter topics.
Click a topic name. On the Consumer Offsets page that is displayed, view the topic consumption details.
View the consumption relationship diagram.

Click Consumers. In the Active Topic area, view all consumer groups in the current cluster and topics that are being consumed by each consumer group.

NOTE:

MRS clusters do not support redirection by clicking a consumer group name.

Parent topic: Using Kafka

Previous topic: Creating a Kafka Topic

Next topic: Managing Kafka Topics

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot