Quickly Using Kafka to Produce and Consume Data

Scenario

You can create, query, and delete topics on a cluster client. Set user permissions by referring to Kafka User Permissions. Once set, perform operations by referring to Using the Kafka Client to Produce and Consume Data (for Versions Earlier Than MRS 3.x).

For clusters of MRS 3.1.2 or later, you can also log in to the Kafka UI to view the consumption information of the current cluster. For details, see Using the Kafka UI to View Consumption Information (for MRS 3.1.2 and Later).

Prerequisites

• If you plan to use the Kafka client, ensure it is installed in a directory (such as /opt/client). The client directory used in the following operations is just an example and should be replaced with the actual installation directory.
• If you plan to use Kafka UI, ensure you have created a user who has the permission to accessit. To perform related operations on the page, for example, creating a topic, you need to grant related permissions to the user. For details, see Kafka User Permissions.

  Site trust must be added to the browser when you access Manager and Kafka UI for the first time. Otherwise, Kafka UI cannot be accessed.

Using the Kafka Client to Produce and Consume Data (for Versions Earlier Than MRS 3.x)

1. Install the client. For details, see Installing a Client.
2. Access the ZooKeeper instance page.
   Click the cluster name to go to the cluster details page and choose Components > ZooKeeper > Instances.

   

   If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

3. View the IP addresses of the ZooKeeper role instance.

   Record any IP address of the ZooKeeper instance.

4. Log in to the node where the client is installed.
5. Run the following command to switch to the client installation directory, for example, /opt/client/Kafka/kafka/bin.

   cd /opt/client/Kafka/kafka/bin

6. Run the following command to configure environment variables:

   source /opt/client/bigdata_env

7. If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.

   kinit Kafka user

8. Create a topic.

   sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

   Example: sh kafka-topics.sh --create --topic TopicTest --partitions 3 --replication-factor 3 --zookeeper 10.10.10.100:2181/kafka

9. Run the following command to view the topic information in the cluster:

   sh kafka-topics.sh --list --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

   Example: sh kafka-topics.sh --list --zookeeper 10.10.10.100:2181/kafka

10. Delete the topic created in 8.

    sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --delete --topic TopicTest --zookeeper 10.10.10.100:2181/kafka

    Type y and press Enter.

Using the Kafka Client to Produce and Consume Data (for MRS 3.x or Later)

1. Install the client. For details, see Installing a Client.
2. Access the ZooKeeper instance page.

   Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Services > ZooKeeper > Instances.

3. View the IP addresses of the ZooKeeper role instance.

   Record any IP address of the ZooKeeper instance.

4. Log in to the node where the client is installed.
5. Run the following command to switch to the client installation directory, for example, /opt/client/Kafka/kafka/bin.

   cd /opt/client/Kafka/kafka/bin

6. Run the following command to configure environment variables:

   source /opt/client/bigdata_env

7. If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.

   kinit Kafka user

8. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper, and click the Configurations tab and then All Configurations. On the displayed page, search for the clientPort parameter and record its value.
9. Create a topic.

   sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

   Example: sh kafka-topics.sh --create --topic TopicTest --partitions 3 --replication-factor 3 --zookeeper 10.10.10.100:2181/kafka

10. Run the following command to view the topic information in the cluster:

    sh kafka-topics.sh --list --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --list --zookeeper 10.10.10.100:2181/kafka

11. Delete the topic created in 9.

    sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --delete --topic TopicTest --zookeeper 10.10.10.100:2181/kafka

Using the Kafka UI to View Consumption Information (for MRS 3.1.2 and Later)

1. Access the Kafka UI.
   1. Log in to FusionInsight Manager as a user who has the permission to access the Kafka UI and choose Cluster > Services > Kafka.

      If you need to perform operations on the page, for example, creating a topic, you need to grant related permissions to users. For details, see Kafka User Permissions.

   2. On the right of KafkaManager WebUI, click the URL to access the Kafka UI.

2. In the Cluster Summary area, view the number of existing topics, brokers, and consumer groups in the current cluster.

   

3. You can click the number under Brokers, Topics, or Consumer Group to go to the corresponding page and view and perform operations on the information.
4. In the Cluster Action area, create topics and migrate partitions. For details, see Adding Kafka Topic Partitions.
5. In the Topic Rank column, view top 10 topics by the number of topic logs, data volume, incoming data volume, and outgoing data volume in the current cluster.

   

6. Click a topic name in the TopicName column to go to the topic details page. For details about operations on the page, see Viewing Kafka Data Production and Consumption Details.