Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Troubleshooting/ Using Kafka/ Flume Normally Connects to Kafka But Fails to Send Messages
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Flume Normally Connects to Kafka But Fails to Send Messages

Symptom

An MRS cluster is installed, and ZooKeeper, Flume, and Kafka are installed in the cluster.

Flume fails to send data to Kafka.

Possible Causes

  1. The Kafka service is abnormal.
  2. The IP address for Flume to connect to Kafka is incorrect.
  3. The size of the message sent from Flume to Kafka exceeds the upper limit.

Cause Analysis

The possible reasons why Flume fails to send data to Kafka may be related to Flume or Kafka.

  1. Check the Kafka service status and monitoring metrics on Manager.

    • MRS Manager: Log in to MRS Manager and choose Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
    • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka.

      Check the Kafka status. It is found that the status is good and the monitoring metrics are correctly displayed.

  2. Check the Flume log. The log contains MessageSizeTooLargeException information, as shown in the following: 
    2016-02-26 14:55:19,126 | WARN  | [SinkRunner-PollingRunner-DefaultSinkProcessor] |  Produce request with correlation id 349829 failed due to [LOG,7]: kafka.common.MessageSizeTooLargeException  | kafka.utils.Logging$class.warn(Logging.scala:83)

    The exception shows that the size of data written to Kafka by Flume exceeds the maximum message size specified by Kafka.

  3. Check the maximum message size specified by Kafka on Manager.
    • MRS Manager portal: Log in to MRS Manager and choose Services > Kafka > Configuration.
    • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka > Configuration.

    On the page that is displayed, set Type to All. All Kafka configurations are displayed. Enter message.max.bytes in the Search text box to search.

    In MRS, the maximum size of a message that can be received by the Kafka server is 1000012 bytes = 977 KB by default.

Solution

After confirmation with the customer, data sent by Flume contains messages over 1 MB. Adjust parameters on Kafka to enable the messages to be written to Kafka.

  1. Set message.max.bytes to a value that is larger than the current maximum size of the message to be written so that Kafka can receive all messages.
  2. Set replica.fetch.max.bytes to a value that is equal to or larger than the value of message.max.bytes so that replicas of partitions on different Brokers can be synchronized to all messages.

    • MRS Manager portal: Log in to MRS Manager and choose Services > Kafka > Configuration.
    • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka > Configuration.

    On the page that is displayed, set Type to All. All Kafka configurations are displayed. Enter replica.fetch.max.bytes in the Search text box to search.

  3. Click Save and restart the Kafka service to make Kafka configurations take effect.
  4. Set fetch.message.max.bytes to a value that is equal to or larger than the value of message.max.bytes for Consumer service applications to ensure that Consumers can consume all messages.
Parent topic: Using Kafka