Setting the Kafka Topic Partition Quantity

Overview

The topic partition quantity is one of the key configurations that affect the Kafka performance, throughput, and concurrency. If the quantity is too small, the concurrency of producers is limited, and consumers may be idle. On the contrary, clusters' performance may deteriorate and monitoring intervals may increase. Therefore, it is important to set a proper partition quantity.

Implementation

In terms of the cluster scale, consumer quantity, and scalability, suggestions are as follows:

**Table 1** Setting the partition quantity
Factor	Suggestion
Cluster scale	Use an integer multiple of the broker quantity. In other cases, partitions on each broker may be uneven, causing unbalanced cluster load.
Consumer quantity	Use the consumer quantity or an integer multiple of it. A partition is consumed by only one consumer. When the partition quantity is less than the consumer quantity, some consumers are idle.
Scalability	For services whose traffic fluctuates greatly, use the consumer quantity during peak hours. Temporarily adding partitions is too slow to troubleshoot stacked messages in existing partitions. Use the consumer quantity during peak hours in advance.

Typical Cases

**Table 2** Typical cases
Symptom	Cause Analysis	Suggestion
ClickHouse is used to consume Kafka. The topic partition quantity is 12, the ClickHouse node quantity is 10, and consumer_num is set to 12 in the ClickHouse parameters (that is, 120 consumers on 10 nodes). Only one ClickHouse node is consuming Kafka, and other ones are idle.	The 12 consumers who are consuming messages are reassigned to all partitions of the topic. These consumers are on the same ClickHouse node. As a result, consumers on other nodes have no message to consume.	Use a partition quantity that is greater than or equal to the consumer quantity. You are advised to set it to an integer multiple of the consumer quantity so that each consumer can be assigned with the same number of partitions for load balancing.
The topic quantity is 3. As the service traffic increases, increase the 3 brokers of the Kafka instance to 6 ones. The new brokers do not have partitions and services are still on the original brokers.	The number of partitions was not increased accordingly and partition reassignment was not performed. As a result, partitions were still on the original brokers.	Use an integer multiple of the broker quantity as the partition quantity. Then, perform partition reassignment to ensure that partitions are even on brokers.