Running a Kafka Job

Updated on 2024-10-11 GMT+08:00

View PDF

You can submit programs developed by yourself to MRS to execute them, and obtain the results. This topic describes how to generate and consume messages in a Kafka topic.

Currently, Kafka jobs cannot be submitted on the GUI. You can submit them in the background.

Submitting a Job in the Background

Query the instance addresses of ZooKeeper and Kafka, and then run the Kafka job.

Querying the Instance Address (3.x)

Log in to the MRS console.
Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
Go to the FusionInsight Manager page. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). On MRS Manager, choose Services > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.
Choose Services > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Querying the Instance Address (Versions Earlier Than 3.x)

Log in to the MRS console.
Choose Clusters > Active Clusters, select a running cluster, and click its name to switch to the cluster details page.
On the MRS cluster details page, choose Components > ZooKeeper > Instance to query the IP addresses of ZooKeeper instances. Record any IP address of a ZooKeeper instance.
Choose Components > Kafka > Instance to query the IP addresses of Kafka instances. Record any IP address of a Kafka instance.

Running a Kafka Job

The default client installation path for MRS 3.x or later is /opt/Bigdata/client, and for versions earlier than MRS 3.x is /opt/client. Configure the path based on site requirements.

Log in to the Master2 node. For details, see Logging In to an ECS.
Run the following command to initialize environment variables:

source /opt/Bigdata/client/bigdata_env
If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step.

kinit MRS cluster user

Example: kinit admin
Run the following command to create a Kafka topic:

kafka-topics.sh --create --zookeeper <IP address of the ZooKeeper role instance:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>
Produce messages in a topic test.

Run the following command: kafka-console-producer.sh --broker-list <IP address of the Kafka role instance:9092> --topic <Topic name> --producer.config /opt/Bigdata/client/Kafka/kafka/config/producer.properties.

Input specified information as the messages produced by the producer and then press Enter to send the messages. To end message production, press Ctrl+C to exit.
Consume messages in the topic test.

kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <Kafka role instance IP:210079092> --consumer.config /opt/Bigdata/client/Kafka/kafka/config/consumer.properties

NOTE:

If Kerberos authentication is enabled in the cluster, change the port number 9092 to 21007 when running the preceding two commands. For details, see List of Open Source Component Ports.

Parent topic: Job Management

Previous topic: Running a Flink Job

Next topic: Viewing Job Configuration and Logs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot