Kafka Consumer Loses Consumed Data

Symptom

A user saves the consumed data to the database and finds that the data is inconsistent with the production data. Therefore, it is suspected that some of Kafka's consumed data is lost.

Possible Causes

• The service code is incorrect.
• An exception occurs when Kafka production data is written.
• The Kafka consumption data is abnormal.

Solution

Check Kafka.

1. Observe the changes of the written and consumed offset through consumer-groups.sh. (Produce a certain number of messages, and consume these messages on the client to observe the changes of the offset.)

   

2. Create a consumption group, use the client to consume messages, and view the consumed messages.

   new-consumer:

   kafka-console-consumer.sh --topic <topic name> --bootstrap-server <IP1:PORT, IP2:PORT,...> --new-consumer --consumer.config <config file>

Check the service code.

1. Check whether an error is reported when the offset is submitted on the client.
2. If no error is reported, add a printing message to the API that is consumed, and print only the key to view the lost data.