Checking Whether Data Consumed by a Customer Is Lost

Symptom

A Customer saves the consumed data to the database and finds that the data is inconsistent with the production data. Therefore, it is suspected that some of Kafka's consumed data is lost.

Possible Causes

• The customer code is incorrect.
• An exception occurs when Kafka production data is written.
• The Kafka consumption data is abnormal.

Solution

Check Kafka.

1. Observe the changes of the written and consumed offset through consumer-groups.sh. (Produce a certain number of messages, and consume these messages on the client to observe the changes of the offset.)

   

2. Create a consumption group, use the client to consume messages, and view the consumed messages.

   new-consumer:

   kafka-console-consumer.sh --topic <topic name> --bootstrap-server <IP1:PORT, IP2:PORT,...> --new-consumer --consumer.config <config file>

Check the customer code.

1. Check whether an error is reported when the offset is submitted on the client.
2. If no error is reported, add a printing message to the API that is consumed, and print only the key to view the lost data.