An Exception Occurs When ClickHouse Consumes Kafka Data

Symptom

A user creates the Kafka engine table test.user_log_kafka in the ClickHouse cluster to consume Kafka data. However, the Kafka monitoring result shows that messages have been stacked since the early morning and data has not been consumed.

Cause Analysis

If ClickHouse encounters an exception in consuming data, messages are stacked in Kafka. You need to check the ClickHouse logs to locate the fault.

1. Log in to the MRS cluster, go to the node where the ClickHouse instance is located, and view the clickhouse-server.log file in the /var/log/Bigdata/clickhouse directory. The following error information is displayed.

   

2. The same error log is found on other ClickHouse nodes. Therefore, Kafka messages are stacked because an exception occurs when ClickHouse parses Kafka data.

Procedure

Run the following command to modify the kafka_skip_broken_messages attribute of the table:

# ALTER test.user_log MODIFY SETTINGS kafka_skip_broken_messages=10000

• Value 10000 in this command can be adjusted based on the proportion of dirty data.
• kafka_skip_broken_messages indicates the tolerance of the Kafka message parser to schema-incompatible messages per block. The default value is 0.

  For example, if kafka_skip_broken_messages is set to N, the engine skips N Kafka messages that cannot be parsed.