Help Center/ MapReduce Service/ Component Operation Guide (ME-Abu Dhabi Region)/ Using Kafka/ Using Kafka from Scratch
Updated on 2024-07-19 GMT+08:00
View PDF
Share

Using Kafka from Scratch

Scenario

You can create, query, and delete topics on a cluster client.

Prerequisites

The client has been installed in a directory, for example, /opt/client. The client directory in the following operations is only an example. Change it based on site requirements.

Using the Kafka Client (Versions Earlier Than MRS 3.x)

  1. Access the ZooKeeper instance page.

    Click the cluster name to go to the cluster details page and choose Components > ZooKeeper > Instances.

    If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

  2. View the IP addresses of the ZooKeeper role instance.

    Record any IP address of the ZooKeeper instance.

  3. Log in to the node where the client is installed.
  4. Run the following command to switch to the client installation directory, for example, /opt/client/Kafka/kafka/bin.

    cd /opt/client/Kafka/kafka/bin

  5. Run the following command to configure environment variables:

    source /opt/client/bigdata_env

  6. If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.

    kinit Kafka user

  7. Create a topic.

    sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --create --topic TopicTest --partitions 3 --replication-factor 3 --zookeeper 10.10.10.100:2181/kafka

  8. Run the following command to view the topic information in the cluster:

    sh kafka-topics.sh --list --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --list --zookeeper 10.10.10.100:2181/kafka

  9. Delete the topic created in 7.

    sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --delete --topic TopicTest --zookeeper 10.10.10.100:2181/kafka

    Type y and press Enter.

Using the Kafka Client (MRS 3.x or Later)

  1. Access the ZooKeeper instance page.

    Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Name of the desired cluster > Services > ZooKeeper > Instance.

  2. View the IP addresses of the ZooKeeper role instance.

    Record any IP address of the ZooKeeper instance.

  3. Log in to the node where the client is installed.
  4. Run the following command to switch to the client installation directory, for example, /opt/client/Kafka/kafka/bin.

    cd /opt/client/Kafka/kafka/bin

  5. Run the following command to configure environment variables:

    source /opt/client/bigdata_env

  6. If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.

    kinit Kafka user

  7. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper, and click the Configurations tab and then All Configurations. On the displayed page, search for the clientPort parameter and record its value.
  8. Create a topic.

    sh kafka-topics.sh --create --topic Topic name --partitions Number of partitions occupied by the topic --replication-factor Number of replicas of the topic --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --create --topic TopicTest --partitions 3 --replication-factor 3 --zookeeper 10.10.10.100:2181/kafka

  9. Run the following command to view the topic information in the cluster:

    sh kafka-topics.sh --list --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --list --zookeeper 10.10.10.100:2181/kafka

  10. Delete the topic created in 8.

    sh kafka-topics.sh --delete --topic Topic name --zookeeper IP address of the node where the ZooKeeper instance resides:clientPort/kafka

    Example: sh kafka-topics.sh --delete --topic TopicTest --zookeeper 10.10.10.100:2181/kafka

Parent topic: Using Kafka