Why Is Partition Quantity Limited?
Kafka manages messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented, affecting the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.
The partition limit varies depending on the flavor, as shown in the following table.
Flavor |
Brokers |
Maximum TPS per Broker |
Maximum Partitions per Broker |
Recommended Consumer Groups per Broker |
Maximum Client Connections per Broker |
Storage Space |
Traffic per Broker (MB/s) |
|---|---|---|---|---|---|---|---|
kafka.2u4g.cluster.small |
3–30 |
20,000 |
100 |
15 |
2000 |
300 GB–300,000 GB |
40 |
kafka.2u4g.cluster |
3–30 |
30,000 |
250 |
20 |
2000 |
300 GB–300,000 GB |
100 |
kafka.4u8g.cluster |
3–30 |
100,000 |
500 |
100 |
4000 |
300 GB–600,000 GB |
200 |
kafka.8u16g.cluster |
3–50 |
150,000 |
1000 |
150 |
4000 |
300 GB–1,500,000 GB |
375 |
kafka.12u24g.cluster |
3–50 |
200,000 |
1500 |
200 |
4000 |
300 GB–1,500,000 GB |
625 |
kafka.16u32g.cluster |
3–50 |
250,000 |
2000 |
200 |
4000 |
300 GB–1,500,000 GB |
750 |
Flavor |
Brokers |
TPS per Broker |
Maximum Partitions per Broker |
Recommended Consumer Groups per Broker |
Maximum Client Connections per Broker |
Storage Space |
Traffic per Broker (MB/s) |
|---|---|---|---|---|---|---|---|
kafka.2u4g.single.small |
1 |
20,000 |
100 |
15 |
2000 |
100 GB–10,000 GB |
40 |
kafka.2u4g.single |
1 |
30,000 |
250 |
20 |
2000 |
100 GB–10,000 GB |
100 |
Topics and Partitions FAQs
- Is There a Limit on the Number of Topics in a Kafka Instance?
- Why Is Partition Quantity Limited?
- Can I Reduce the Partition Quantity?
- Why Do I Fail to Create Topics?
- Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?
- Why Do Deleted Topics Still Exist?
- Can I View the Disk Space Used by a Topic?
- Can I Add ACL Permissions for Topics?
- What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
- How Do I Increase the Partition Quantity?
- Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modified?
- Can I Delete Unnecessary Topics in a Consumer Group?
- What Can I Do If a Consumer Fails to Retrieve Messages from a Topic Due to Insufficient Permissions?
- Why Does an Instance Contain Default Topics __trace and __consumer_offsets?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore
