Cluster Kafka Instances

Instance Specifications

A cluster Kafka instance has three or more brokers, and is compatible with open-source Kafka 1.1.0, 2.7, and 3.x.

For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second. In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB. The test scenario is private access in plaintext. The disk type is ultra-high I/O. For more information about TPS performance, see Kafka Instance TPS.

Instance Specifications and Network Bandwidth

The network bandwidth of a Kafka instance consists of the following:

1. Network bandwidth used by the instance brokers
2. Bandwidth of the disk used by the instance brokers. For details, see Disk Types and Performance.

Note:

• By default, Kafka tests are performed in the tail read scenario (that is, only the latest production data is consumed) instead of the cold read scenario (that is, historical data is consumed from the beginning).
• The bandwidth of an instance with an old flavor (such as 100 MB/s) is the total network bandwidth of the instance's all brokers.

Traffic calculation of instances with new flavors (such as kafka.2u4g.cluster) is described as follows:

• The read/write ratio is 1:1.
• The default number of topic replicas is 3.
• Total network traffic = Traffic per broker x Broker quantity
• Total instance traffic = Service traffic + Data replication traffic between brokers

Assume that the current flavor is kafka.2u4g.cluster, the traffic per broker is 100 MB/s, and the number of brokers is 3. What are the total network traffic, maximum read traffic, and maximum write traffic of the instance?

1. Total network traffic = Traffic per broker x Broker quantity = 100 MB/s x 3 = 300 MB/s
2. Maximum read traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2= 50 MB/s
3. Maximum write traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2 = 50 MB/s

Mapping Between Old and New Flavors

Table 2 compares the old and new Kafka instance flavors.

Instances with new flavors have the following features:

• Better performance and cost effectiveness: They use exclusive resources (except for kafka.2u4g.cluster.small). By contrast, old flavors use non-exclusive resources. If the load is heavy, resources conflicts will occur.
• Latest functions, for example, reassigning partitions, changing the SSL setting, and viewing rebalancing logs.
• Flexible flavor changes: For example, you can increase or decrease the broker flavor.
• Flexible disk capacity: Only related to the broker quantity, and not to the flavor.
• More specification options: A wider range of combinations of broker flavor (over 10,000 MB/s) and quantity are available.
• More disk type options: General Purpose SSD and Extreme SSD are now available, in addition to the original disk types.

Flavor Selection

• kafka.2u4g.cluster.small with 3 brokers

  Recommended for up to 6000 client connections, 45 consumer groups, and 60,000 TPS

• kafka.2u4g.cluster with 3 brokers

  Recommended for up to 6000 client connections, 60 consumer groups, and 90,000 TPS

• kafka.4u8g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 300 consumer groups, and 300,000 TPS

• kafka.8u16g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 450 consumer groups, and 450,000 TPS

• kafka.12u24g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 600 consumer groups, and 600,000 TPS

• kafka.16u32g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 600 consumer groups, and 750,000 TPS

Storage Space Selection

Kafka instances can store messages in multiple replicas. The storage space is consumed by message replicas, logs, and metadata. When creating an instance, specify its storage space based on the expected service message size, the number of replicas, and reserved disk space. Each Kafka broker reserves 33 GB disk space for storing logs and metadata.

For example, if the expected service message size is 100 GB, the number of replicas is 2, and the number of brokers is 3, the disk size should be at least 299 GB (100 GB x 2 + 33 GB x 3).

The storage space can be expanded as your service grows.

Topic Quantity

There are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Figure 1) and the specified number of partitions in each topic (see Table 1).

Figure 1 Setting the number of partitions

The maximum number of partitions allowed for an instance with kafka.2u4g.cluster and 3 brokers is 750.

• If the number of partitions of each topic in the instance is 3, the maximum number of topics is 750/3 = 250.
• If the number of partitions of each topic in the instance is 1, the maximum number of topics is 750/1 = 750.