Producer Occasionally Fails to Send Data and the Log Displays "Too many open files in system"

Updated on 2022-12-08 GMT+08:00

View PDF

Symptom

When Producer sends data to Kafka, it is found that the client fails to send data.

Figure 1 Producer fails to send data.

Possible Causes

The Kafka service is abnormal.
The network is abnormal.
The Kafka topic is abnormal.

Cause Analysis

Check the Kafka service status:
- MRS Manager: Log in to MRS Manager and choose Services > Kafka. Check the Kafka status. The status is Good, and the monitoring metrics are correctly displayed.
- FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Name of the target cluster > Service > Kafka.
  Check the Kafka status. It is found that the status is good and the monitoring metrics are correctly displayed.
View the error topic information in the SparkStreaming log.
Run the Kafka commands to obtain the topic assignment information and copy synchronization information, and check the return result.

kafka-topics.sh --describe --zookeeper <zk_host:port/chroot>

As shown in Figure 2, the topic status is normal. All partitions have normal leader information.

Figure 2 Topic status
Run the telnet command to check whether the Kafka can be connected.
telnet Kafka service IP address Kafka service port

If telnet fails, check the network security group and ACL.
Log in to Kafka Broker using SSH.
Run the cd /var/log/Bigdata/kafka/broker command to go to the log directory.

Check on server.log indicates that the error message is displayed in the log shown in the following figure.

Figure 3 Log exception
Output of the lsof command used to check the handle usage of the Kafka process on the current node shows that the number of handles used by the Kafka process reaches 470,000.
Figure 4 Handles
Check the service code. It is found that the Producer object is frequently created and is not closed normally.

Solution

Stop the current application to ensure that the number of handles on the server does not increase sharply, which affects the normal running of services.
Optimize the application code to resolve the handle leakage problem.

Suggestion: Use one Producer object globally. After the use is complete, call the Close interface to close the handle.

Parent topic: Using Kafka

Previous topic: Producer Fails to Send Data and "TOPIC_AUTHORIZATION_FAILED" Is Thrown

Next topic: Consumer Is Initialized Successfully, But the Specified Topic Message Cannot Be Obtained from Kafka

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot