On this page

Show all

Help Center/ MapReduce Service/ Troubleshooting/ Using Kafka/ Flume Normally Connects to Kafka but Fails to Send Messages

Flume Normally Connects to Kafka but Fails to Send Messages

Updated on 2023-11-30 GMT+08:00

View PDF

Symptom

An MRS cluster is installed, and ZooKeeper, Flume, and Kafka are installed in the cluster.

Flume fails to send data to Kafka.

Possible Causes

The Kafka service is abnormal.
The IP address for Flume to connect to Kafka is incorrect.
The size of the message sent from Flume to Kafka exceeds the upper limit.

Cause Analysis

The possible reasons why Flume fails to send data to Kafka may be related to Flume or Kafka.

Check the Kafka service status and monitoring metrics on Manager.
- MRS Manager: Log in to MRS Manager and choose Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
- FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.

Check the Flume log. The log contains MessageSizeTooLargeException information, as shown in the following:

2016-02-26 14:55:19,126 | WARN  | [SinkRunner-PollingRunner-DefaultSinkProcessor] |  Produce request with correlation id 349829 failed due to [LOG,7]: kafka.common.MessageSizeTooLargeException  | kafka.utils.Logging$class.warn(Logging.scala:83)

The exception shows that the size of data written to Kafka by Flume exceeds the maximum message size specified by Kafka.

Check the maximum message size specified by Kafka on Manager.
- MRS Manager: Log in to MRS Manager and choose Services > Kafka > Configurations.
- FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Services > Kafka > Configurations.
On the page that is displayed, set Type to All. All Kafka configurations are displayed. Enter message.max.bytes in the Search text box to search.

In MRS, the maximum size of a message that can be received by the Kafka server is 977 KB (1000012 bytes) by default.

Solution

After confirmation with the customer, data sent by Flume contains messages over 1 MB. Adjust parameters on Kafka to enable the messages to be written to Kafka.

Set message.max.bytes to a value that is larger than the current maximum size of the message to be written so that Kafka can receive all messages.
Set replica.fetch.max.bytes to a value that is equal to or larger than the value of message.max.bytes so that replicas of partitions on different Brokers can be synchronized to all messages.
- MRS Manager: Log in to MRS Manager and choose Services > Kafka > Configurations.
- FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Services > Kafka > Configurations.
On the page that is displayed, set Type to All. All Kafka configurations are displayed. Enter replica.fetch.max.bytes in the Search text box to search.
Click Save and restart the Kafka service to make Kafka configurations take effect.
Set fetch.message.max.bytes to a value that is equal to or larger than the value of message.max.bytes for Consumer service applications to ensure that Consumers can consume all messages.

Parent topic: Using Kafka

Previous topic: Using Python3.x to Connect to Kafka in a Security Cluster

Next topic: Producer Fails to Send Data and Error Message "NullPointerException" Is Displayed

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot