Kafka Broker Reports Abnormal Processes and the Log Shows "IllegalArgumentException"

Symptom

The Process Fault alarm is reported on Manager. Check whether the faulty process is Kafka Broker.

Possible Causes

Broker configuration is abnormal.

Cause Analysis

1. On Manager, obtain the host information on the alarm page.
2. Log in to Kafka Broker using SSH. Run the cd /var/log/Bigdata/kafka/broker command to go to the log directory.

   Check the server.log file. It is found that the "IllegalArgumentException" exception is thrown in the following log stating "java.lang.IllegalArgumentException: requirement failed: replica.fetch.max.bytes should be equal or greater than message.max.bytes."

   2017-01-25 09:09:14,930 | FATAL | [main] |  | kafka.Kafka$ (Logging.scala:113)
   java.lang.IllegalArgumentException: requirement failed: replica.fetch.max.bytes should be equal or greater than message.max.bytes
           at scala.Predef$.require(Predef.scala:233)
           at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:959)
           at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:944)
           at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:701)
           at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:698)
           at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
           at kafka.Kafka$.main(Kafka.scala:60)
           at kafka.Kafka.main(Kafka.scala)

   Kafka requires that replica.fetch.max.bytes be greater than or equal to message.max.bytes.

3. On the Kafka configuration page, select All Configurations. All Kafka configurations are displayed. Search for message.max.bytes and

   replica.fetch.max.bytes. It is found that the value of replica.fetch.max.bytes is less than that of message.max.bytes.

Solution

1. Go to the Kafka configuration page.
   • For versions earlier than MRS 3.x: Log in to MRS Manager and choose Services > Kafka > Service Configuration > All Configurations.
   • For MRS 3.x or later: Log in to FusionInsight Manager and choose Cluster > Services > Kafka > Configurations > All Configurations.

2. Search for and modify the replica.fetch.max.bytes parameter to ensure that its value is greater than or equal to that of message.max.bytes. In this way, replicas of partitions on different brokers can be synchronized to all messages.
3. Save the configuration and check whether there is any service whose configuration has expired in the cluster. If yes, restart the corresponding service or role instance for the configuration to take effect.
4. Modify fetch.message.max.bytes in the Consumer service application to ensure that the value of fetch.message.max.bytes is greater than or equal to that of message.max.bytes.