Updated on 2024-05-30 GMT+08:00

Why Is Partition Quantity Limited?

Kafka manages messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented, affecting the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.

The partition limit varies depending on the flavor, as shown in the following table.

Table 1 Kafka instance specifications (cluster)

Flavor

Brokers

Maximum TPS per Broker

Maximum Partitions per Broker

Recommended Consumer Groups per Broker

Maximum Client Connections per Broker

Storage Space

Traffic per Broker (MB/s)

kafka.2u4g.cluster

3–30

30,000

250

20

2000

300 GB–300,000 GB

100

kafka.4u8g.cluster

3–30

100,000

500

100

4000

300 GB–600,000 GB

200

kafka.8u16g.cluster

3–50

150,000

1000

150

4000

300 GB–1,500,000 GB

375

kafka.12u24g.cluster

3–50

200,000

1500

200

4000

300 GB–1,500,000 GB

625

kafka.16u32g.cluster

3–50

250,000

2000

200

4000

300 GB–1,500,000 GB

750