Kafka Consumer Reads Oversized Records

Updated on 2022-12-14 GMT+08:00

View PDF

Symptom

After data is written to Kafka, a user develops an application and invokes the interface (org.apache.kafka.clients.consumer.*) to read data from Kafka as a Consumer. However, the reading fails and the following error is reported:

..........
1687 [KafkaConsumerExample] INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Successfully joined group DemoConsumer with generation 1
1688 [KafkaConsumerExample] INFO  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator  - Setting newly assigned partitions [default-0, default-1, default-2] for group DemoConsumer
2053 [KafkaConsumerExample] ERROR com.xxxxxx.bigdata.kafka.example.NewConsumer  - [KafkaConsumerExample], Error due to 
org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {default-0=177} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size on the client (using max.partition.fetch.bytes), or decrease the maximum message size the broker will allow (using message.max.bytes).
2059 [KafkaConsumerExample] INFO  com.xxxxxx.bigdata.kafka.example.NewConsumer  - [KafkaConsumerExample], Stopped 
.......

Cause Analysis

When reading data, the Kafka client compares the size of the data to be read with the value of max.partition.fetch.bytes. If the size exceeds the value of max.partition.fetch.bytes, the preceding exception is reported.

Solution

When creating a Kafka Consumer instance during initialization, set max.partition.fetch.bytes.

For example, you can set this parameter to 5252880 as follows:

......
// Security protocol type
props.put(securityProtocol, kafkaProc.getValues(securityProtocol, "SASL_PLAINTEXT"));
// Service name
props.put(saslKerberosServiceName, "kafka");
        
props.put("max.partition.fetch.bytes","5252880");
......

Parent topic: Using Kafka

Previous topic: Kafka Producer Writes Oversized Records

Next topic: High Usage of Multiple Disks on a Kafka Cluster Node

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot