Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Kafka/ Using the Kafka Client
Updated on 2024-12-11 GMT+08:00
View PDF

Using the Kafka Client

Scenario

This section guides users to use a Kafka client in an O&M or service scenario.

This section applies to MRS 3.x or later.

Prerequisites

  • The cluster client has been installed in a directory, for example, /opt/client.
  • Service users of each component are created by the MRS cluster administrator based on service requirements. A machine-machine user needs to download the keytab file and a human-machine user needs to change the password upon the first login. (Not involved in normal mode)
  • After changing the domain name of a cluster, redownload the client to ensure that the kerberos.domain.name value in the configuration file of the client is set to the correct server domain name.

Using a Kafka Client

  1. Log in to the node where the client is installed as the client installation user.
  2. Run the following command to go to the client installation directory:

    cd /opt/client

  3. Run the following command to configure environment variables:

    source bigdata_env

  4. Run the following command to perform user authentication (skip this step in normal mode):

    kinit Component service user

  5. Run the following command to switch to the Kafka client installation directory:

    cd Kafka/kafka/bin

  6. Run the following command to use the client tool to view and use the help information:

    • ./kafka-console-consumer.sh: Kafka message reading tool
    • ./kafka-console-producer.sh: Kafka message publishing tool
    • ./kafka-topics.sh: Kafka topic management tool

  7. To perform various operations on Kafka topics for versions prior to MRS 3.x, execute the appropriate commands.

    • Command for creating a topic:

      sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    • Command for deleting a topic:

      sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    • The number of topic partitions or topic backup copies cannot exceed the number of Kafka instances.
    • By default, the ZooKeeper's clientPort value is 2181.
    • There are three ZooKeeper instances. Use the IP address of any one.

  8. MRS 3.x and later versions: Use kafka-topics.sh to manage Kafka topics.

    • Creating a topic:

      ./kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic--replication-factor Number of backups of the topic --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka

      ./kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of backups of the topic --bootstrap-server Kafka Cluster IP address:21007 --command-config ../config/client.properties

    • Listing topics:
      • ./kafka-topics.sh --list --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
      • ./kafka-topics.sh --list --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties
    • Viewing a topic:
      • ./kafka-topics.sh --describe --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --topic Topic name
      • ./kafka-topics.sh --describe --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties --topic Topic name
    • Modifying a topic:
      • ./kafka-topics.sh --alter --topic Topic name --config Configuration item=Value--zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
    • Expanding partitions:
      • ./kafka-topics.sh --alter --topic Topic name --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka --command-config Kafka/kafka/config/client.properties --partitions Number of partitions after the expansion
      • ./kafka-topics.sh --alter --topic Topic name --bootstrap-server IP address of the Kafka cluster:21007 --command-config Kafka/kafka/config/client.properties --partitions Number of partitions after the expansion
    • Deleting a topic:
      • ./kafka-topics.sh --delete --topic Topic name --zookeeper Service IP address of any ZooKeeper node:clientPort/kafka
      • ./kafka-topics.sh --delete --topic Topic name --bootstrap-server IP address of the Kafka cluster:21007 --command-config ../config/client.properties

Parent topic: Using Kafka