Specifications

Kafka Instance Specifications

Kafka instances are compatible with open-source Kafka v1.1.0, v2.3.0, and v2.7. The instance specifications are represented by the ECS flavor. Available options are kafka.2u4g.cluster, kafka.4u8g.cluster, kafka.8u16g.cluster, kafka.12u24g.cluster, and kafka.16u32g.cluster.

For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second. In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB.

Instance Specifications and Network Bandwidth

The network bandwidth of a Kafka instance consists of the following:

1. Network bandwidth used by the instance brokers
2. Bandwidth of the disk used by the instance brokers. For details, see Disk Types and Performance.

Note:

• By default, Kafka tests are performed in the tail read scenario (that is, only the latest production data is consumed) instead of the cold read scenario (that is, historical data is consumed from the beginning).
• The bandwidth of an instance with an old flavor (such as 100 MB/s) is the total network bandwidth of the instance's all brokers.

Traffic calculation of instances with new flavors (such as kafka.2u4g.cluster) is described as follows:

• The read/write ratio is 1:1.
• The default number of topic replicas is 3.
• Total network traffic = Traffic per broker x Broker quantity
• Total instance traffic = Service traffic + Data replication traffic between brokers

Assume that the current flavor is kafka.2u4g.cluster, the traffic per broker is 100 MB/s, and the number of brokers is 3. What are the total network traffic, maximum read traffic, and maximum write traffic of the instance?

1. Total network traffic = Traffic per broker x Broker quantity = 100 MB/s x 3 = 300 MB/s
2. Maximum read traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2= 50 MB/s
3. Maximum write traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2 = 50 MB/s

Mapping Between Old and New Flavors

Table 2 compares the old and new Kafka instance flavors.

Instances with new flavors have the following features:

• Better performance and cost effectiveness: They use exclusive resources. By contrast, old flavors use non-exclusive resources. If the load is heavy, resources conflicts will occur.
• Latest functions, for example, reassigning partitions.
• Flexible flavor changes: For example, you can increase or decrease the broker flavor.
• Flexible disk capacity: Only related to the broker quantity, and not to the flavor.
• More specification options: A wider range of combinations of broker flavor (over 10,000 MB/s) and quantity are available.

Flavor Selection

• kafka.2u4g.cluster with 3 brokers

  Recommended for up to 6000 client connections, 60 consumer groups, and 90,000 TPS

• kafka.4u8g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 300 consumer groups, and 300,000 TPS

• kafka.8u16g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 450 consumer groups, and 450,000 TPS

• kafka.12u24g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 600 consumer groups, and 600,000 TPS

• kafka.16u32g.cluster with 3 brokers

  Recommended for up to 12,000 client connections, 600 consumer groups, and 750,000 TPS

Storage Space Selection

Kafka instances support multi-replica storage. The storage space is consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved space).

The storage space can be expanded as your service grows.

Topic Quantity

There are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Figure 1) and the specified number of partitions in each topic (see Table 1).

Figure 1 Setting the number of partitions

The maximum number of partitions allowed for an instance with kafka.2u4g.cluster and 3 brokers is 750.

• If the number of partitions of each topic in the instance is 3, the maximum number of topics is 750/3 = 250.
• If the number of partitions of each topic in the instance is 1, the maximum number of topics is 750/1 = 750.