Using the Kafka Client

Updated on 2024-12-11 GMT+08:00

View PDF

Scenario

This section guides users to use a Kafka client in an O&M or service scenario.

This section applies to MRS 3.x or later.

Prerequisites

The cluster client has been installed in a directory, for example, /opt/client.
Service users of each component are created by the MRS cluster administrator based on service requirements. A machine-machine user needs to download the keytab file and a human-machine user needs to change the password upon the first login. (Not involved in normal mode)
After changing the domain name of a cluster, redownload the client to ensure that the kerberos.domain.name value in the configuration file of the client is set to the correct server domain name.

Using a Kafka Client

Log in to the node where the client is installed as the client installation user.
Run the following command to go to the client installation directory:

cd /opt/client
Run the following command to configure environment variables:

source bigdata_env
Run the following command to perform user authentication (skip this step in normal mode):

kinit Component service user
Run the following command to switch to the Kafka client installation directory:

cd Kafka/kafka/bin
Run the following command to use the client tool to view and use the help information:
- ./kafka-console-consumer.sh: Kafka message reading tool
- ./kafka-console-producer.sh: Kafka message publishing tool
- ./kafka-topics.sh: Kafka topic management tool
To perform various operations on Kafka topics for versions prior to MRS 3.x, execute the appropriate commands.
- Command for creating a topic:
  sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka
- Command for deleting a topic:
  sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka
NOTE:
- The number of topic partitions or topic backup copies cannot exceed the number of Kafka instances.
- By default, the ZooKeeper's clientPort value is 2181.
- There are three ZooKeeper instances. Use the IP address of any one.
MRS 3.x and later versions: Use kafka-topics.sh to manage Kafka topics.
- Creating a topic:
  ./kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic--replication-factor Number of backups of the topic --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
  
  ./kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of backups of the topic --bootstrap-server Kafka Cluster IP address:21007 --command-config ../config/client.properties
- Listing topics:
  - ./kafka-topics.sh --list --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
  - ./kafka-topics.sh --list --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties
- Viewing a topic:
  - ./kafka-topics.sh --describe --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --topic Topic name
  - ./kafka-topics.sh --describe --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties --topic Topic name
- Modifying a topic:
  - ./kafka-topics.sh --alter --topic Topic name --config Configuration item=Value--zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
- Expanding partitions:
  - ./kafka-topics.sh --alter --topic Topic name --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --command-config Kafka/kafka/config/client.properties --partitions Number of partitions after the expansion
  - ./kafka-topics.sh --alter --topic Topic name --bootstrap-server IP address of the Kafka cluster:21007 --command-config Kafka/kafka/config/client.properties --partitions Number of partitions after the expansion
- Deleting a topic:
  - ./kafka-topics.sh --delete --topic Topic name --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
  - ./kafka-topics.sh --delete --topic Topic name --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties