Accessing Messages in Kafka Topics

Scenario

You can use the Kafka client or Kafka UI to view the current consumption information based on service requirements.

This section applies to MRS 3.x or later.

Prerequisites

When using the Kafka client, ensure that the following conditions are met:

• The MRS cluster administrator has understood service requirements and prepared a system user.
• The Kafka client has been installed.

Using the Kafka Client to View the Consumption Information

1. Log in as a client installation user to the node on which the Kafka client is installed.
2. Switch to the Kafka client installation directory, for example, /opt/client.

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Run the following command to perform user authentication (skip this step in normal mode):

   kinit Component service user

5. Run the following command to switch to the Kafka client installation directory:

   cd Kafka/kafka/bin

6. Run the kafka-consumer-groups.sh command to check the current consumption status.
   • Check the Consumer Group list on Kafka saved by Offset:

     ./kafka-consumer-groups.sh --list --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --command-config ../config/consumer.properties

     Example: ./kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:21007 --list --command-config ../config/consumer.properties

   • Check the consumption status of Consumer Group on Kafka saved by Offset:

     ./kafka-consumer-groups.sh --describe --bootstrap-server <Service IP address of any node where a broker instances is located:Port number of the Kafka cluster> --group Consumer group name --command-config ../config/consumer.properties

     Example: ./kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.1:21007 --group example-group --command-config ../config/consumer.properties

   

   • Ensure that the current consumer is online and consumes data.
   • Configure the group.id in the consumer.properties configuration file and --group in the command to the group to be queried.
   • The Kafka cluster's IP port number is 21007 in security mode and 9092 in normal mode.

Using KafkaUI to View the Current Consumption Status

1. Access the Kafka UI.
   1. Log in to FusionInsight Manager as a user who has the permission to access the Kafka UI and choose Cluster > Services > Kafka.

      If you need to perform related operations on the page, for example, creating a topic, you need to grant related permissions to the user. For details, see Kafka User Permissions.

   2. On the right of KafkaManager WebUI, click the URL to access Kafka UI.

2. Click Consumers. On the consumer group details page that is displayed, you can view all consumer groups in the current cluster and the IP address of the node where each consumer group coordinator is located. In the upper right corner of the page, you can enter a consumer group name to search for the specified consumer group.

   

3. In the Consumer Summary area, you can view the existing consumer groups in the current cluster. You can click a consumer group name to view the topics consumed by the consumer group. Consumed topics can be in the pending or running state. pending indicates that the topic has been consumed but not being consumed. running indicates that the topic is being consumed. You can enter a topic name in the upper right corner of the dialog box to filter topics.

   

4. Click a topic name. On the Consumer Offsets page that is displayed, view the topic consumption details.

   

5. View the consumption lineage chart.

   Click Consumers. The consumer group details page is displayed. In the Active Topic area, view all consumer groups in the current cluster and topics that are being consumed by each consumer group.

   

   

   MRS clusters do not support redirection by clicking a consumer group name.