Help Center > > User Guide> Managing an Existing Cluster> Job Management> Running a Kafka Job

Running a Kafka Job

Updated at: Mar 25, 2021 GMT+08:00

You can submit programs developed by yourself to MRS to execute them, and obtain the results. This topic describes how to generate and consume messages in a Kafka topic.

Currently, Kafka jobs cannot be submitted on the GUI. You can submit them in the background.

Submitting a Job in the Background

Query the instance addresses of ZooKeeper and Kafka, and then run the Kafka job.

Querying the Instance Address (3.x Version)

  1. Log in to the MRS management console.
  2. Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
  3. Go to the FusionInsight Manager page. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). On MRS Manager, choose Services > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.
  4. Choose Services > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Querying the Instance Address (2.x and Earlier Versions)

  1. Log in to the MRS management console.
  2. Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
  3. On the MRS cluster details page, choose Components > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.

    For MRS 1.8.10 or earlier, log in to MRS Manager. For details, see Accessing MRS Manager (MRS 2.1.0 or Earlier). Choose Services > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.

  4. Choose Components > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Running a Kafka Job

  1. On the Nodes tab page of the cluster detail page, click the name of the Master2 node to go to the ECS management console.
  2. Click Remote Login in the upper right corner of the page.
  3. Enter the username and password of the Master node as prompted. The username is root and the password is the one set during cluster creation.
  4. Run the following command to initialize environment variables:

    source /opt/client/bigdata_env

  5. If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step.

    kinit MRS cluster user

    Example: kinit admin

  6. Run the following command to create a Kafka topic:

    kafka-topics.sh --create --zookeeper <IP address of the ZooKeeper role instance:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>

  7. Produce messages in a topic test.

    Run the following command: kafka-console-producer.sh --broker-list <IP address of the Kafka role instance:9092> --topic <Topic name> --producer.config /opt/client/Kafka/kafka/config/producer.properties.

    Input specified information as the messages produced by the producer and then press Enter to send the messages. To end message production, press Ctrl+C to exit.

  8. Consume messages in the topic test.

    kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <IP address of the Kafka role instance:9092> --new-consumer --consumer.config /opt/client/Kafka/kafka/config/consumer.properties

    If Kerberos authentication is enabled in the cluster, change the port number 9092 to 21007 when running the preceding two commands. For details, see List of Open Source Component PortsList of Open Source Component Ports.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel